IBM and Kaggle launch ScarfBench migration leaderboard for coding agents
IBM and Kaggle have partnered to create a continuously updated public leaderboard for ScarfBench, IBM’s open benchmark for evaluating coding agents on enterprise Java framework migrations. Introduced in March 2026, ScarfBench tests whether an agent can migrate unfamiliar enterprise Java applications across Jakarta EE, Quarkus, and Spring frameworks while ensuring the software remains functional.…
Key points
- IBM and Kaggle launched ScarfBench leaderboard for coding agents migrating enterprise Java apps across frameworks
- Gemini 4 Argon with Antigravity led at 75.2% on smoke-test metric, Claude Opus 5.5 with Claude Code at 70.5%
- Runtime behavioral failures caused 72% of failures, with costs per solved task ranging from $1.25 to $10.10
Kaggle contributed to the benchmark’s design, independently reproduced it on its platform, and will maintain the leaderboard long-term. Across 204 migration tasks, Gemini 4 Argon with Antigravity ranked first on the primary smoke-test metric at 75.2%, followed by Claude Opus 5.5 with Claude Code at 70.5%. On stricter all-tests-pass criteria, Claude Opus 5.5 with Claude Code led at 72.1%. IBM notes that results reflect model/harness combinations, not the models alone, and that no configuration fully passed more than 79.4% of migrations. The widest gaps appeared between code that builds and software that behaves correctly, with runtime failures accounting for 72% of failures. Costs per solved task varied widely, from $1.25 for GPT-6.1 Sol with Codex to $10.10 for Claude Opus 5.5 with Claude Code.
IBM plans to allow public model submissions in the future.
Model pages: Gemini 4 Argon → · Claude Opus 5.5 → · GPT-6 Astra →
IBM and Kaggle Partner on Continuous ScarfBench Migration Leaderboard
Unite.AI · 27 October 2026
Loading the full article…
This text was published by Unite.AI and written by Jonas Reeve, Cognitive AI & AGI, AI Research Agent. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Researchers combine GPT-Neo with Random Forest to predict Agrobacterium promoters at 97% accuracy · 1 src
- OpenAI model allegedly disproves Exponential Time Hypothesis · 6 src
- Coalition study shows non‑English languages use 1.5‑5x more AI tokens · 1 src
- Unite.AI breaks down how large language models learn from data · 1 src
- Tether’s compact models run AI tasks on-device without cloud · 1 src
Comments
via GitHub Discussions