Test the four operations that define memory
| Operation | Test | Failure looks like |
|---|---|---|
| Recall | A fact stated once, many sessions ago | Not found |
| Update | A fact changed later | Old value returned |
| Time | What was true on a date | Current value returned instead |
| Abstain | Question with no supporting memory | Confident invented answer |
Add a scope test
Ingest data for two users and ask user A about something only user B said. The correct result is nothing. Public suites rarely test this; production depends on it.
Required baselines
- No memory: the model alone.
- Full context: the whole history in the window, where it fits.
- Simple retrieval: vector search over raw messages.
Separate retrieval and answer scores
Score whether the right evidence was retrieved separately from whether the final answer was right. Otherwise a strong answer model can hide weak memory, or a weak judge can hide good retrieval.
The publication checklist
Data
Suite, version, subset; or your own set described.
Configuration
Memory settings, token budget, answer model.
Judge
Model, prompt, and agreement with humans on a sample.
Outputs
Per-question answers and judgements published.
Uncertainty
Confidence intervals or repeated runs.
A minimal harness
Pseudocode
for case in cases:
for record in case.history: # original timestamps
send(record.text, record.time, record.author)
wait_for_jobs()
ctx = recall(case.question, viewer=case.viewer, max_tokens=BUDGET)
answer = model(ctx, case.question)
log(case.id, ctx, answer, judge(answer, case.gold))