{"version":1,"type":"story","url":"https://digestai.news/story/ibm-and-kaggle-launch-scarfbench-migration-leaderboard-for-coding-agen","json":"https://digestai.news/story/ibm-and-kaggle-launch-scarfbench-migration-leaderboard-for-coding-agen.json","markdown":"https://digestai.news/story/ibm-and-kaggle-launch-scarfbench-migration-leaderboard-for-coding-agen.md","slug":"ibm-and-kaggle-launch-scarfbench-migration-leaderboard-for-coding-agen","headline":"IBM and Kaggle launch ScarfBench migration leaderboard for coding agents","summary":"IBM and Kaggle have partnered to create a continuously updated public leaderboard for **ScarfBench**, IBM’s open benchmark for evaluating coding agents on enterprise Java framework migrations. Introduced in March 2026, ScarfBench tests whether an agent can migrate unfamiliar enterprise Java applications across **Jakarta EE, Quarkus, and Spring frameworks** while ensuring the software remains functional. The benchmark includes developer-verified workloads, automated validation, and behavioral tests to measure success rates.\n\nKaggle contributed to the benchmark’s design, independently reproduced it on its platform, and will maintain the leaderboard long-term. Across **204 migration tasks**, **Gemini 4 Argon with Antigravity** ranked first on the primary smoke-test metric at **75.2%**, followed by **Claude Opus 5.5 with Claude Code** at **70.5%**. On stricter all-tests-pass criteria, **Claude Opus 5.5 with Claude Code** led at **72.1%**. IBM notes that results reflect model/harness combinations, not the models alone, and that no configuration fully passed more than **79.4%** of migrations. The widest gaps appeared between code that builds and software that behaves correctly, with runtime failures accounting for **72%** of failures. Costs per solved task varied widely, from **$1.25** for **GPT-6.1 Sol with Codex** to **$10.10** for **Claude Opus 5.5 with Claude Code**.\n\nIBM plans to allow public model submissions in the future.","keyPoints":["IBM and Kaggle launched ScarfBench leaderboard for coding agents migrating enterprise Java apps across frameworks","Gemini 4 Argon with Antigravity led at 75.2% on smoke-test metric, Claude Opus 5.5 with Claude Code at 70.5%","Runtime behavioral failures caused 72% of failures, with costs per solved task ranging from $1.25 to $10.10"],"whyItMatters":"ScarfBench provides a rigorous, real-world benchmark for evaluating coding agents in enterprise migrations, helping developers and AI labs compare performance and efficiency across frameworks. The leaderboard’s public nature encourages transparency and competition in AI-driven software modernization.","category":{"slug":"research","name":"Research","url":"https://digestai.news/category/research"},"entities":{"companies":["IBM","Kaggle"],"models":["Gemini 4 Argon","Claude Opus 5.5","Claude Opus 5","GPT-6 Astra","GPT-6.1 Sol","Kimi K3"],"people":[]},"firstPublishedAt":"2026-10-27T12:18:01Z","updatedAt":"2026-10-27T12:18:01Z","sourceCount":1,"hasPrimarySource":false,"sources":[{"outlet":"Unite.AI","title":"IBM and Kaggle Partner on Continuous ScarfBench Migration Leaderboard","url":"https://unite.ai/ibm-and-kaggle-partner-on-continuous-scarfbench-migration-leaderboard","publishedAt":"2026-10-27T12:18:01Z","type":"press","primary":false,"lead":true}],"sourceNotes":null,"discussions":[],"thread":null,"cite":{"text":"Digest AI, \"IBM and Kaggle launch ScarfBench migration leaderboard for coding agents\", 27 October 2026, https://digestai.news/story/ibm-and-kaggle-launch-scarfbench-migration-leaderboard-for-coding-agen","publisher":"Digest AI","title":"IBM and Kaggle launch ScarfBench migration leaderboard for coding agents","datePublished":"2026-10-27T12:18:01Z","url":"https://digestai.news/story/ibm-and-kaggle-launch-scarfbench-migration-leaderboard-for-coding-agen"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}