# Collinear AI releases CWE-bench v2 with 100 open-source cybersecurity tasks

Digest AI · Research · published 2026-11-03T00:00:00Z

Canonical: https://digestai.news/story/collinear-ai-releases-cwe-bench-v2-with-100-open-source-cybersecurity

## Summary

Collinear AI has published **CWE-bench v2**, an updated benchmark for testing AI models’ ability to detect and fix cybersecurity vulnerabilities in real open-source code. The new version expands coverage to **91 different CWE categories** across 100 tasks, each designed to evaluate a model’s capacity to identify weaknesses without breaking existing functionality. Tasks are scored programmatically, ensuring reproducibility, and include a held-out set for tracking progress on unsolved problems. The benchmark is open-source, with results tracked on Hugging Face Hub via pull requests for both open-weight and API-only models like **Gemini 4 Argon** (67.8% pass@1) and **Claude Opus 5.5** (67.3% pass@1). The leaderboard also highlights cost-efficiency, with **DeepSeek V4.1 Flash** achieving 57.0% pass@1 at **$0.04 per rollout**, while **Gemini 4 Argon** leads at **$3.92** for higher accuracy. The dataset is Dockerized for easy execution via **Harbor**, and contributors can submit feedback or results directly.

## Key points

- CWE-bench v2 covers 91 CWEs across 100 open-source tasks, each testing AI’s ability to fix vulnerabilities without breaking code
- Gemini 4 Argon leads with 67.8% pass@1, but even top models fail 19% of tasks consistently
- Open-weight models can submit results via PRs on Hugging Face Hub; API-only models require a separate leaderboard Space

## Why it matters

This benchmark pushes AI cybersecurity testing beyond memorization, requiring models to generalize across diverse vulnerability types. It provides a standardized way for labs to compare defensive AI capabilities and could accelerate research into automated security tools.

## Sources

1. [CWE-bench v2: More coverage, quality, and transparency](https://huggingface.co/blog/cwe-bench) (Hugging Face, 2026-11-03, primary source)

Part of the developing story: [Mistral's Rise in Open AI Development](https://digestai.news/thread/mistral-and-nvidia-collaborate-to-integrate-ai-models-into-engineering-workflows) (3 stories)

## Cite

Digest AI, "Collinear AI releases CWE-bench v2 with 100 open-source cybersecurity tasks", 3 November 2026, https://digestai.news/story/collinear-ai-releases-cwe-bench-v2-with-100-open-source-cybersecurity

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/collinear-ai-releases-cwe-bench-v2-with-100-open-source-cybersecurity.json