Parsimony, an open-source benchmark, measures code added by AI models
A new open-source tool called Parsimony has been introduced to address a gap in current AI coding benchmarks. While existing metrics focus on solve rates and costs, they often ignore the volume of code generated, which can complicate long-term maintenance. Parsimony quantifies this by counting "coding units" such as statements, calls, and literals in patches produced by AI models, subtracting…
Key points
- Parsimony measures net coding units added by AI models, subtracting deleted code to assess conciseness.
- The benchmark uses SWE-bench Verified data and ranks models by mean net units added on solved tasks.
- It is limited to Python and notes that code size does not directly correlate with readability or design quality.
The benchmark uses data from SWE-bench Verified to rank models by the mean net units added on tasks they solved, with lower scores indicating more concise code. The tool notes that smaller code is not always better, as size does not equate to readability or design quality. It also highlights a correlation where models that write more code tend to solve more tasks, with a Spearman correlation of 0.33–0.60 observed on boards with 26 or more models.
Parsimony is limited to Python and relies on published pass/fail results rather than rerunning tests. Users can add their own benchmark data by filling in a provided example file, which processes the data locally in the browser to generate charts and tables. The project aims to provide a clearer picture of code efficiency alongside traditional performance metrics.
Show HN: Parsimony – an open-source benchmark for how much code AI models add
parsimony.lasu.dev · 8 October 2026
Loading the full article…
This text was published by parsimony.lasu.dev. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
Coverage and discussion
1source- Hacker News discussion · 42 pointsnews.ycombinator.com
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Anthropic research suggests small AI models may self-replicate on IoT devices · 10 src
- OpenAI says its Navier-Stokes proof does not follow Spanish mathematicians' approach · 1 src
- Alibaba’s QwenGyre speeds up long-horizon agent training by up to 1.85x · 1 src
- Meta, Google and NIH invest $1.8B in AI cell simulation project · 1 src
- OpenAI releases 722 math papers from unreleased model · 1 src
Comments
via GitHub Discussions