{"version":1,"type":"story","url":"https://digestai.news/story/collinear-ai-releases-cwe-bench-v2-with-100-open-source-cybersecurity","json":"https://digestai.news/story/collinear-ai-releases-cwe-bench-v2-with-100-open-source-cybersecurity.json","markdown":"https://digestai.news/story/collinear-ai-releases-cwe-bench-v2-with-100-open-source-cybersecurity.md","slug":"collinear-ai-releases-cwe-bench-v2-with-100-open-source-cybersecurity","headline":"Collinear AI releases CWE-bench v2 with 100 open-source cybersecurity tasks","summary":"Collinear AI has published **CWE-bench v2**, an updated benchmark for testing AI models’ ability to detect and fix cybersecurity vulnerabilities in real open-source code. The new version expands coverage to **91 different CWE categories** across 100 tasks, each designed to evaluate a model’s capacity to identify weaknesses without breaking existing functionality. Tasks are scored programmatically, ensuring reproducibility, and include a held-out set for tracking progress on unsolved problems. The benchmark is open-source, with results tracked on Hugging Face Hub via pull requests for both open-weight and API-only models like **Gemini 4 Argon** (67.8% pass@1) and **Claude Opus 5.5** (67.3% pass@1). The leaderboard also highlights cost-efficiency, with **DeepSeek V4.1 Flash** achieving 57.0% pass@1 at **$0.04 per rollout**, while **Gemini 4 Argon** leads at **$3.92** for higher accuracy. The dataset is Dockerized for easy execution via **Harbor**, and contributors can submit feedback or results directly.","keyPoints":["CWE-bench v2 covers 91 CWEs across 100 open-source tasks, each testing AI’s ability to fix vulnerabilities without breaking code","Gemini 4 Argon leads with 67.8% pass@1, but even top models fail 19% of tasks consistently","Open-weight models can submit results via PRs on Hugging Face Hub; API-only models require a separate leaderboard Space"],"whyItMatters":"This benchmark pushes AI cybersecurity testing beyond memorization, requiring models to generalize across diverse vulnerability types. It provides a standardized way for labs to compare defensive AI capabilities and could accelerate research into automated security tools.","category":{"slug":"research","name":"Research","url":"https://digestai.news/category/research"},"entities":{"companies":["Collinear AI","Hugging Face"],"models":["Gemini 4 Argon","Claude Opus 5.5","DeepSeek V4.1 Flash"],"people":[]},"firstPublishedAt":"2026-11-03T00:00:00Z","updatedAt":"2026-11-03T00:00:00Z","sourceCount":1,"hasPrimarySource":true,"sources":[{"outlet":"Hugging Face","title":"CWE-bench v2: More coverage, quality, and transparency","url":"https://huggingface.co/blog/cwe-bench","publishedAt":"2026-11-03T00:00:00Z","type":"primary","primary":true,"lead":true}],"sourceNotes":null,"discussions":[],"thread":{"title":"Mistral's Rise in Open AI Development","url":"https://digestai.news/thread/mistral-and-nvidia-collaborate-to-integrate-ai-models-into-engineering-workflows","storyCount":3},"cite":{"text":"Digest AI, \"Collinear AI releases CWE-bench v2 with 100 open-source cybersecurity tasks\", 3 November 2026, https://digestai.news/story/collinear-ai-releases-cwe-bench-v2-with-100-open-source-cybersecurity","publisher":"Digest AI","title":"Collinear AI releases CWE-bench v2 with 100 open-source cybersecurity tasks","datePublished":"2026-11-03T00:00:00Z","url":"https://digestai.news/story/collinear-ai-releases-cwe-bench-v2-with-100-open-source-cybersecurity"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}