Megagon Labs releases mawile workbench for auditing LLM judges
MAWILE is a new developer‑facing workbench that lets researchers and engineers audit the sensitivity of large‑language‑model (LLM) judges. LLM judges are increasingly used to score model and agent outputs, but their verdicts can shift when irrelevant details of the prompt, rubric, input, or output change. MAWILE addresses this by taking a user‑provided judge and a set of evaluation items, then…
Key points
- MAWILE audits LLM judges across prompt, rubric, input, and output perturbations
- It validates controlled changes and localizes sensitivity without needing gold labels
- The open‑source code is available at github.com/megagonlabs/mawile-judge
The tool works with binary, ordinal, and pairwise judges and does not require gold‑standard labels, making it practical for a wide range of evaluation pipelines. MAWILE’s code is released openly on GitHub, allowing the community to adopt and extend the framework for more reliable LLM benchmarking.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Study finds LLM judge consensus overstates evidence because judges share errors · 1 src
- Study shows LLM deliberation improves accuracy across domains · 1 src
- Study finds Gemini 3.1 Flash-Lite deficit recurs on fresh data under historical configuration · 1 src
- Researchers propose Goal-driven variant categorization using LLM · 1 src
- Study shows AI agents select fewer papers when they see others' choices · 1 src
Comments
via GitHub Discussions