Collinear AI releases CWE-bench v2 with 100 open-source cybersecurity tasks
Collinear AI has published CWE-bench v2, an updated benchmark for testing AI models’ ability to detect and fix cybersecurity vulnerabilities in real open-source code. The new version expands coverage to 91 different CWE categories across 100 tasks, each designed to evaluate a model’s capacity to identify weaknesses without breaking existing functionality. Tasks are scored programmatically,…
Key points
- CWE-bench v2 covers 91 CWEs across 100 open-source tasks, each testing AI’s ability to fix vulnerabilities without breaking code
- Gemini 4 Argon leads with 67.8% pass@1, but even top models fail 19% of tasks consistently
- Open-weight models can submit results via PRs on Hugging Face Hub; API-only models require a separate leaderboard Space
Model pages: Gemini 4 Argon → · Claude Opus 5.5 →
The story so far
3 episodes →- Collinear AI releases CWE-bench v2 with 100 open-source cybersecurity tasksthis story
CWE-bench v2: More coverage, quality, and transparency
Hugging Face · 3 November 2026
Loading the full article…
This text was published by Hugging Face and written by Gonzalo Gonzalez. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Anthropic releases FlashProve for cryptographic AI inference proofs · 3 src
- MIT researchers report GPT-6 Astra natively reads and writes neural network weights, raising Qwen3.5-0.8B MATH-500 from 7.9% to 48.1% by editing 0.016% of weigh · 6 src
- SETI’s Allen Telescope Array uses AI agents to speed up signal analysis · 1 src
- Stanford study: LLMs' claims expire quickly · 1 src
- Python script separates basketball shot selection from making · 1 src
Comments
via GitHub Discussions