# IBM and Kaggle launch ScarfBench migration leaderboard for coding agents

Digest AI · Research · published 2026-10-27T12:18:01Z

Canonical: https://digestai.news/story/ibm-and-kaggle-launch-scarfbench-migration-leaderboard-for-coding-agen

## Summary

IBM and Kaggle have partnered to create a continuously updated public leaderboard for **ScarfBench**, IBM’s open benchmark for evaluating coding agents on enterprise Java framework migrations. Introduced in March 2026, ScarfBench tests whether an agent can migrate unfamiliar enterprise Java applications across **Jakarta EE, Quarkus, and Spring frameworks** while ensuring the software remains functional. The benchmark includes developer-verified workloads, automated validation, and behavioral tests to measure success rates.

Kaggle contributed to the benchmark’s design, independently reproduced it on its platform, and will maintain the leaderboard long-term. Across **204 migration tasks**, **Gemini 4 Argon with Antigravity** ranked first on the primary smoke-test metric at **75.2%**, followed by **Claude Opus 5.5 with Claude Code** at **70.5%**. On stricter all-tests-pass criteria, **Claude Opus 5.5 with Claude Code** led at **72.1%**. IBM notes that results reflect model/harness combinations, not the models alone, and that no configuration fully passed more than **79.4%** of migrations. The widest gaps appeared between code that builds and software that behaves correctly, with runtime failures accounting for **72%** of failures. Costs per solved task varied widely, from **$1.25** for **GPT-6.1 Sol with Codex** to **$10.10** for **Claude Opus 5.5 with Claude Code**.

IBM plans to allow public model submissions in the future.

## Key points

- IBM and Kaggle launched ScarfBench leaderboard for coding agents migrating enterprise Java apps across frameworks
- Gemini 4 Argon with Antigravity led at 75.2% on smoke-test metric, Claude Opus 5.5 with Claude Code at 70.5%
- Runtime behavioral failures caused 72% of failures, with costs per solved task ranging from $1.25 to $10.10

## Why it matters

ScarfBench provides a rigorous, real-world benchmark for evaluating coding agents in enterprise migrations, helping developers and AI labs compare performance and efficiency across frameworks. The leaderboard’s public nature encourages transparency and competition in AI-driven software modernization.

## Sources

1. [IBM and Kaggle Partner on Continuous ScarfBench Migration Leaderboard](https://unite.ai/ibm-and-kaggle-partner-on-continuous-scarfbench-migration-leaderboard) (Unite.AI, 2026-10-27)

## Cite

Digest AI, "IBM and Kaggle launch ScarfBench migration leaderboard for coding agents", 27 October 2026, https://digestai.news/story/ibm-and-kaggle-launch-scarfbench-migration-leaderboard-for-coding-agen

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/ibm-and-kaggle-launch-scarfbench-migration-leaderboard-for-coding-agen.json