Researchers release German court benchmark for LLM legal reasoning
A new arXiv paper introduces a sentence-level benchmark to test how well large language models can classify interpretive canons used by the German Federal Constitutional Court. The dataset consists of court decisions annotated at the sentence level, based on a legal theory from Larenz in the Savigny tradition. The authors evaluated four LLMs from three model families using both expert-written…
1 source primary source
Key points
- New sentence-level benchmark uses German Federal Constitutional Court decisions
- Four LLMs from three families scored 70.4-79.2 mean F1 across seven canons
- GEPA-optimized prompts did not systematically beat expert hand-written prompts
Read the original at arXiv cs.CL · by Felix Ringe primary sourceOpen source ↗
Topics · follow one to build your own front page
LarenzSavigny
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Anthropic's Claude begins 'dreaming' to reorganize memory in tests · 1 src
- John Langford quits Microsoft, declares AI Frontiers over · 1 src
- Anthropic releases study on Claude usage in scientific research · 3 src
- MIT and BASF release CheMeleon for low-data molecular prediction · 2 src
- Anthropic tests Claude agents in EVE Online trading and sabotage detection · 1 src
Comments
via GitHub Discussions