arXiv study finds open-weight models match closed models in CTI-REALM tasks
The authors tested nineteen open-weight and seven closed models under the same conditions, scoring their ability to convert reports into working detections by querying telemetry data.
Key points
- Nineteen open-weight and seven closed models tested on **CTI-REALM** benchmark for threat-intelligence agents
- Open-weight models matched closed models’ performance when given more interaction turns, per paper
- Stage-level metrics show different models excel at distinct workflow stages, per authors
The paper found that open-weight models can match the best closed models when given more interaction turns. Removing this limit revealed that open-weight models outperformed closed ones in later stages. The study also introduced stage-level metrics to evaluate each model’s specific strengths, suggesting workflows could route tasks to the best-performing model per step rather than relying on a single model’s overall score. Token efficiency varied widely, with no clear link between cost and performance.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Asset learns to compress speech encoders · 1 src
- Researchers find Qwen3’s reasoning gains may stem from ‘thinking leakage’ · 1 src
- Researchers test AI agents improving LLM training strategies with up to 13.34% gains · 1 src
- Researchers model how AI and humans time new words in text · 1 src
- Researchers propose Next-Concept Modeling for faster text generation · 1 src
Comments
via GitHub Discussions