Researchers release IntLawNER dataset for international law NER
The paper introduces IntLawNER, a new named‑entity‑recognition dataset covering 2,987 gold‑annotated sentences and 8,094 entity spans drawn from International Court of Justice decisions, UN Security Council resolutions, and European Court of Human Rights judgments. The data use seven institution‑specific entity types and were built with a hybrid algorithmic‑agentic pipeline that trimmed 468 k…
Key points
- IntLawNER contains 2,987 gold‑annotated sentences and 8,094 entity spans from ICJ, UN Security Council, and ECtHR texts.
- The hybrid pipeline reduced 468 k source sentences to the final set, with 89.6% of gold spans unchanged from the silver layer.
- Claude Opus 4.6 achieved the highest benchmark score, 0.873 micro‑F1, while zero‑shot GLiNER scored only 0.243 micro‑F1.
Benchmark results show that aggregate agreement metrics can be misleading: Cohen’s kappa reaches 0.964 on boundary‑matched spans, yet macro‑F1 drops to 0.753 when missing entities, boundary errors, and label corrections are considered. Zero‑shot GLiNER collapses on function‑based entity types with only 0.243 micro‑F1, while fine‑tuned transformers struggle with rare labels. Providing carefully selected few‑shot examples improves all LLMs, and Claude Opus 4.6 attains the highest score of 0.873 micro‑F1. The authors release IntLawNER as a reusable benchmark for extracting references in international legal texts.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Researchers propose multi-split boundary decision to lower LLM document segmentation cost · 1 src
- GaitVista reduces gait measurement error by 27.7% in lab tests · 1 src
- Megagon Labs releases mawile workbench for auditing LLM judges · 1 src
- AutoGym framework generates verifiable agent gyms · 1 src
- EvidenT improves enterprise assistant evidence verification by 29% · 1 src
Comments
via GitHub Discussions