{"version":1,"type":"story","url":"https://digestai.news/story/researchers-release-bilal-bench-to-test-bengali-ai-models-islamic-text","json":"https://digestai.news/story/researchers-release-bilal-bench-to-test-bengali-ai-models-islamic-text.json","markdown":"https://digestai.news/story/researchers-release-bilal-bench-to-test-bengali-ai-models-islamic-text.md","slug":"researchers-release-bilal-bench-to-test-bengali-ai-models-islamic-text","headline":"Researchers release BILAL-BENCH to test Bengali AI models’ Islamic text fabrication","summary":"A new benchmark called **BILAL-BENCH** tests how open-source AI models generate fabricated Islamic texts in Bengali. Published on arXiv, it includes 240 expert-verified questions covering Qur’anic verses, hadiths, legal rulings, and modern issues. The benchmark evaluates seven open-weight models—ranging from 1 billion to 120 billion parameters—on 1,680 double-annotated responses, scoring them on source accuracy and confidence in claims.\n\nThe results show alarming trends: six of the seven models fabricated responses on **79.3% to 95.3%** of source-production questions. Hadith provenance was the weakest category, with only **2 of 315 responses** correct and properly sourced. Even models with the same size (8B BanglaLLaMA variants) exhibited different error patterns. Notably, models varied widely in whether they avoided unsupported claims—abstention rates ranged from **6 to 46 responses**—but this did not correlate with model size.","keyPoints":["BILAL-BENCH tests Bengali AI models’ fabrication of Islamic texts with 240 expert-verified questions","Six of seven models fabricated responses on 79.3%–95.3% of source-production questions, per double-annotated data","Hadith provenance was the weakest category: only 2 of 315 responses were correct and properly sourced"],"whyItMatters":"This benchmark highlights how open-source AI models may misrepresent sacred texts, undermining trust in AI-generated content for religious or legal contexts. It also shows that model size alone doesn’t guarantee accuracy in specialized domains.","category":{"slug":"research","name":"Research","url":"https://digestai.news/category/research"},"entities":{"companies":[],"models":["BanglaLLaMA"],"people":[]},"firstPublishedAt":"2026-11-02T05:00:00Z","updatedAt":"2026-11-02T05:00:00Z","sourceCount":1,"hasPrimarySource":true,"sources":[{"outlet":"arXiv cs.CL","title":"Do Open Language Models Invent Scripture? BILAL-BENCH: Benchmarking Islamic-Source Fabrication in Bengali","url":"https://arxiv.org/abs/2610.40996","publishedAt":"2026-11-02T05:00:00Z","type":"primary","primary":true,"lead":true}],"sourceNotes":null,"discussions":[],"thread":null,"cite":{"text":"Digest AI, \"Researchers release BILAL-BENCH to test Bengali AI models’ Islamic text fabrication\", 2 November 2026, https://digestai.news/story/researchers-release-bilal-bench-to-test-bengali-ai-models-islamic-text","publisher":"Digest AI","title":"Researchers release BILAL-BENCH to test Bengali AI models’ Islamic text fabrication","datePublished":"2026-11-02T05:00:00Z","url":"https://digestai.news/story/researchers-release-bilal-bench-to-test-bengali-ai-models-islamic-text"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}