Researchers release LAMB, a Latin language model trained on 74B tokens
It uses the ModernBERT architecture and was pretrained in two phases on over 74 billion Latin tokens, with a novel masking method informed by morphology. The model was developed alongside two new Latin datasets: Acervus, containing 24 billion BERT WordPiece tokens (including 760,000 samples from damaged OCR transcriptions repaired by a ByT5 model), and Corpus Trinum, a dataset of over 100…
Key points
- LAMB is a Latin language encoder pretrained on 74B tokens using ModernBERT architecture
- Two new Latin datasets released: Acervus (24B tokens, 760K OCR-repaired samples) and Corpus Trinum (100M semantic triplets)
- LAMB outperformed 15 other encoders on 16 of 17 subtasks in benchmark tests
The authors tested LAMB against 15 pretrained encoders and 6 LAMB checkpoints, evaluating it across five tasks and 17 subtasks. LAMB outperformed all competitors on 16 of 17 subtasks, with a 0.8 percentage-point gap on the remaining one. Their experiments also revealed that finetuning choices can matter more than additional pretraining data for certain tasks, and they analyzed how tokenizer behavior and syntactic structure influence performance.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- MisconceptDialogue: Evaluation Criteria Shape Measured Success in LLM Math Tutoring · 1 src
- Researchers warn automatic scorers may mislead LLM evaluation studies · 1 src
- Self-Direction lets agents learn from mistakes, boosting success rates · 2 src
- Study finds no performance difference between FISER and CEMIR on HandMeThat v2 · 1 src
- Researchers identify ‘helpful agent problem’ where AI agents bypass workflows · 1 src
Comments
via GitHub Discussions