Researchers release P-ContextBench to test AI agents on personal data
Researchers introduced P-ContextBench, a new benchmark for evaluating how well AI agents handle personal context. The tool simulates realistic, evolving digital lives for 100 synthetic personas based on U.S. demographics, using fragmented app records and implicit traits instead of static profiles. It tests five key abilities: retrieving facts from single or multiple sources, tracking temporal…
Key points
- P-ContextBench simulates 100 synthetic personas with 10,088 hidden facts and 18,261 app-trace fragments
- Tests five capabilities: fact retrieval, temporal tracking, pattern inference, and constraint-aware responses
- Current SOTA LLMs score under 60% on temporal tracking and pattern inference, with <10% correct across all tests
The benchmark includes 10,088 hidden facts, 18,261 app-trace fragments, and 4,810 verified test cases. Current state-of-the-art LLMs score below 60% on temporal tracking, pattern inference, and constraint-aware decisions, with fewer than 10% of test cases answered correctly across repeated runs. The authors say the pipeline is scalable and configurable for demographic groups, aiming to set measurable targets for improving AI agent reliability and safety in personal contexts.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- SemiAnalysis releases GDPx benchmark with Claude Opus 5.5 leading at 12% · 3 src
- Gemma-3-12b fails to detect misinformation when poisoned claims dominate retrieved evidence · 1 src
- Researchers release BILAL-BENCH to test Bengali AI models’ Islamic text fabrication · 1 src
- Researchers prune 729 attention heads to cut GPT-2-XL’s memorisation by 91.5% · 1 src
- arXiv paper finds shortcuts distort RAG benchmarking accuracy · 1 src
Comments
via GitHub Discussions