Researchers introduce MemPABench to evaluate personal agent memory
A new benchmark called MemPABench has been released to test how personal AI agents remember and act on users' interaction preferences. The environment simulates four grounded users, each with 14 work‑ and personal‑context preferences, generating 99 sessions over four months and including five event‑driven preference changes per user.
Key points
- MemPABench simulates four users with 14 preferences each across 99 sessions over four months.
- Four memory systems on four LLM backbones were tested; retrieval‑based methods did not beat a no‑memory baseline.
- File‑based memory improved accuracy but often applied preferences beyond the relevant context.
The authors evaluated four memory systems built on four large‑language‑model backbones. Retrieval‑based approaches such as Mem0, MemOS, and simple RAG showed no advantage over a no‑memory baseline, while a file‑based memory improved selection accuracy but often applied preferences outside the relevant context. Overall, the study finds that current memory mechanisms provide limited gains in preference selection and struggle to adapt to changing user preferences, even when the correct preference is available.
The benchmark and dataset are publicly available at https://mempabench.com for further research on reducing interaction burden in personal intelligence systems.
The story so far
2 episodes →- Researchers introduce MemPABench to evaluate personal agent memorythis story
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- MIT historian says early AI engineers added randomness to mimic human creativity · 1 src
- Researchers introduce MemPlan for text-based planning with PDDL · 1 src
- Researchers present CiteAnchor framework for detecting hallucinated citations · 1 src
- Researchers find self‑rewarded moral argumentation makes language models decisive but pliable · 1 src
- AQL-Lite improves Arabic LLMs with curated data and partial-parameter training · 1 src
Comments
via GitHub Discussions