What BEAM measures
BEAM comes from “Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs” by Mohammad Tavakoli, Alireza Salemi, Carrie Ye, Mohamed Abdalla, Hamed Zamani and J Ross Mitchell (arXiv:2510.27246). The authors argue that existing benchmarks often lack narrative coherence, cover narrow domains and test only simple recall.
- A generation framework produces long, coherent, topically diverse conversations of up to 10M tokens.
- Each conversation comes with probing questions that target a wide range of memory abilities.
- The benchmark has 100 conversations and 2,000 validated questions.
- The same paper proposes LIGHT, a method that gives a model three memory systems: long-term episodic memory, short-term working memory and a scratchpad for salient facts.
The paper reports that even models with 1M-token context windows, with or without retrieval augmentation, struggle as dialogues lengthen.
Reading results by history size
The most useful view is accuracy by conversation length: how much does a system degrade as history grows past what a context window can hold? A flat curve suggests retrieval and reconciliation scale; a steep drop suggests reliance on the window.
How to read a leaderboard
- Check that every row used the same answer model and judge.
- Check which lengths were run; some entries skip the largest.
- Prefer results from papers or with published outputs.
- Look at scores per ability, not only the average.
Citing BEAM results
Link the paper (arXiv:2510.27246) and say which conversation lengths and which version of the data you used, so readers can compare like with like.