Researchers release SlackBench to test AI agents on real teamwork tasks
A team of researchers has introduced SlackBench, a new benchmark designed to evaluate AI agents on collaborative work tasks grounded in real-world scenarios. Unlike existing benchmarks that rely on synthetic or isolated tasks, SlackBench simulates complex, multi-party workflows where agents must synthesize information from conversations, code repositories, and evolving decisions—such as those in…
Key points
- SlackBench tests AI agents on 140 queries simulating real teamwork tasks across conversations, code, and evolving decisions
- Top-performing system scores 80%, but most agents score below 66%, showing gaps in collaborative reasoning
- Dataset includes privacy-preserving reenactments of real academic research projects with GitHub and Google Docs data
The benchmark tests 33 frontier models and agent frameworks, with the top-performing system scoring around 80% and most scoring below 66%. This highlights significant room for improvement in AI agents’ ability to handle realistic collaborative environments. SlackBench is available on Hugging Face and includes privacy-preserving reenactments of real research projects, providing a reproducible framework for further development.
The story so far
2 episodes →- Researchers release SlackBench to test AI agents on real teamwork tasksthis story
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Anthropic uses Claude to map bacteriophage vault protein at 3.2 Å resolution · 3 src
- RAGCompass routes retrievers by reranking passages, improves answer F1 by 28% · 1 src
- MIT researchers develop uq-daam to correct robot memory · 1 src
- Qwen improves LLM-based OCR correction with veto gate · 1 src
- Study: LLMs prefer to answer over clarifying ambiguous references · 1 src
Comments
via GitHub Discussions