# Researchers benchmark eight LLMs on English and Bengali summarization

Digest AI · Research · published 2026-10-15T04:00:00Z

Canonical: https://digestai.news/story/researchers-benchmark-eight-llms-on-english-and-bengali-summarization

## Summary

A new study on arXiv compares eight large language models—both closed-source and open-weight—on multilingual abstractive summarization. The authors tested them on size-matched English and Bengali datasets from the XL-Sum benchmark, using lexical metrics (ROUGE, BLEU), semantic evaluations, and reference-free factuality checks. They also used an LLM-as-a-judge approach to assess quality and reliability across dimensions.

Key findings show GPT-5.6 Luna outperforms others in English lexical metrics but struggles in Bengali, where LLaMA 3.3 70B tops ROUGE and BLEU scores. Gemini 2.5 Flash leads in factual consistency for Bengali. The study warns that performance in high-resource languages like English does not guarantee success in low-resource ones like Bengali, stressing the need for diverse evaluation methods.

## Key points

- Eight LLMs tested on English and Bengali summarization using ROUGE, BLEU, factuality checks, and LLM-as-judge scoring
- GPT-5.6 Luna leads English lexical metrics but lags in Bengali, where LLaMA 3.3 70B scores highest
- Gemini 2.5 Flash achieves highest factuality score for Bengali, highlighting cross-lingual performance gaps

## Why it matters

The study exposes limitations in current AI summarization models for low-resource languages, urging developers to prioritize multilingual benchmarks beyond English dominance.

## Sources

1. [Evaluating Large Language Models for Multilingual Abstractive Summarization: A Comprehensive Study of English and Bengali](https://arxiv.org/abs/2610.19003) (arXiv cs.CL, 2026-10-15, primary source)

## Cite

Digest AI, "Researchers benchmark eight LLMs on English and Bengali summarization", 15 October 2026, https://digestai.news/story/researchers-benchmark-eight-llms-on-english-and-bengali-summarization

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/researchers-benchmark-eight-llms-on-english-and-bengali-summarization.json