Researchers release BILAL-BENCH to test Bengali AI models’ Islamic text fabrication
A new benchmark called BILAL-BENCH tests how open-source AI models generate fabricated Islamic texts in Bengali. Published on arXiv, it includes 240 expert-verified questions covering Qur’anic verses, hadiths, legal rulings, and modern issues. The benchmark evaluates seven open-weight models—ranging from 1 billion to 120 billion parameters—on 1,680 double-annotated responses, scoring them on…
Key points
- BILAL-BENCH tests Bengali AI models’ fabrication of Islamic texts with 240 expert-verified questions
- Six of seven models fabricated responses on 79.3%–95.3% of source-production questions, per double-annotated data
- Hadith provenance was the weakest category: only 2 of 315 responses were correct and properly sourced
The results show alarming trends: six of the seven models fabricated responses on 79.3% to 95.3% of source-production questions. Hadith provenance was the weakest category, with only 2 of 315 responses correct and properly sourced. Even models with the same size (8B BanglaLLaMA variants) exhibited different error patterns. Notably, models varied widely in whether they avoided unsupported claims—abstention rates ranged from 6 to 46 responses—but this did not correlate with model size.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- SemiAnalysis releases GDPx benchmark with Claude Opus 5.5 leading at 12% · 3 src
- Researchers release Qistas protocol for testing LLM fairness across cultures · 1 src
- Researchers compare speculative drafters and continuous diffusion language models for text generation · 1 src
- Researchers propose Li²SAO to improve asynchronous RL training for LLMs · 1 src
- Researchers study how LLMs assign emotional value to outcomes · 1 src
Comments
via GitHub Discussions