Researchers release AgentCIScope to test LLM agents’ data privacy risks
Researchers introduced AgentCIScope, a new benchmark platform to evaluate how LLM-powered agents handle sensitive user data across real-world applications. The tool tests agents on 936 executable tasks spanning email, calendars, and messaging, scoring them on whether they comply with privacy policies, user consent, and predefined rules for handling personal data like credit cards or Social…
Key points
- AgentCIScope tests 12 LLM agents on 936 tasks across email, calendars, and messaging apps
- Leakage rates range from 32.5% to 94.4% per task, rising further with repeated attempts
- Personal data like preferences and bookings leak far more often than credit cards or SSNs
The study found that even capable agents leak sensitive information at rates between 32.5% and 94.4% per task, with leakage worsening when tasks are repeated. Personal preferences and booking details were most frequently exposed, while credit-card data and SSNs were handled more securely. The work highlights a gap between an agent’s ability to complete tasks and its adherence to privacy constraints, offering a framework for developers to build safer systems.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- OpenAI offers U.S. university labs free ChatGPT Business seats with Rosalind access · 2 src
- Anthropic reports small AI models may self-replicate on IoT devices within months · 29 src
- Researchers propose LLM-referenced C-test for evaluating supra-intelligence · 8 src
- Researchers study permutation diffusion LLMs for recommendation reranking · 1 src
- Researchers compare Non-Generative verifier to Astra LLM for RAG claim checks · 1 src
Comments
via GitHub Discussions