What MemoryBench is
Its authors argue that most memory benchmarks reduce the problem to long-form reading comprehension, which is a poor proxy for adapting to feedback. MemoryBench instead runs multi-task, multi-domain, multilingual evaluation with simulated user feedback, in both off-policy settings (replaying pre-recorded dialogues) and on-policy settings (generating dialogues on the fly).
- 28 datasets across three domains (Academic & Knowledge, Legal, Open-Domain) and four task shapes.
- Eight memory-system baselines behind one registry interface, including vanilla, BM25 and embedding retrieval, A-Mem, Mem0 and MemoryOS.
- Four experiment regimes: off-policy, stepwise off-policy, on-policy and training-set performance.
- A simulated user-feedback model, plus an LLM-as-judge branch that gives every dataset a unified score from 0 to 1.
- The paper (arXiv:2510.17281) was accepted to ICML 2026 as a spotlight.
Sources: THUIR/MemoryBench repository (README) · MemoryBench paper, arXiv:2510.17281 · Dataset
Where it fits among memory benchmarks
| Kind | Examples | Question asked |
|---|---|---|
| Conversation QA | LoCoMo, LongMemEval, BEAM | Can the system answer from a long history? |
| Feedback and continual learning | MemoryBench | Does the system get better from feedback while in service? |
How to compare MemoryBench scores
- Same datasets and domains, and the same regime (off-policy or on-policy).
- Same metric — automatic metrics or the LLM-judge score — since the repository offers both.
- Same feedback simulator version.
- Same answer model and context budget.
MemoryBench and Auld
No Auld result is published on this site. Running Auld through MemoryBench would mean adding it as a memory-system entry in the benchmark’s registry and reporting the regime, metric and simulator used.