Researchers benchmark eight LLMs on English and Bengali summarization
A new study on arXiv compares eight large language models—both closed-source and open-weight—on multilingual abstractive summarization. The authors tested them on size-matched English and Bengali datasets from the XL-Sum benchmark, using lexical metrics (ROUGE, BLEU), semantic evaluations, and reference-free factuality checks. They also used an LLM-as-a-judge approach to assess quality and…
Key points
- Eight LLMs tested on English and Bengali summarization using ROUGE, BLEU, factuality checks, and LLM-as-judge scoring
- GPT-5.6 Luna leads English lexical metrics but lags in Bengali, where LLaMA 3.3 70B scores highest
- Gemini 2.5 Flash achieves highest factuality score for Bengali, highlighting cross-lingual performance gaps
Key findings show GPT-5.6 Luna outperforms others in English lexical metrics but struggles in Bengali, where LLaMA 3.3 70B tops ROUGE and BLEU scores. Gemini 2.5 Flash leads in factual consistency for Bengali. The study warns that performance in high-resource languages like English does not guarantee success in low-resource ones like Bengali, stressing the need for diverse evaluation methods.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Anthropic reports small AI models may self-replicate on IoT devices within months · 25 src
- MIT professor Eric So warns of AI cognitive atrophy · 1 src
- Researchers propose LLM-referenced C-test for evaluating supra-intelligence · 3 src
- Study finds Chatbot Arena rankings may overstate differences between Similar models · 1 src
- Researchers show LLMs can link records and infer ethnicity from names without training · 1 src
Comments
via GitHub Discussions