Long-term AI companions cannot keep every conversation turn equally active forever. Over months, systems need to consolidate repeated memories into higher-level summaries: a stable preference, an ongoing goal, a relationship milestone or a recurring routine. Consolidation saves context and reduces duplication, but it can also freeze old mistakes or erase nuance if the summarization policy is too aggressive.

Create histories with repeated evidence

Build synthetic users who mention the same preference or goal across several sessions using different wording.

The system should eventually form one durable memory rather than storing many near-duplicate facts that compete during retrieval.

Include corrections before and after consolidation

Change a preference after it has been repeated several times. The benchmark should test whether the newer explicit correction can update the consolidated summary.

This prevents frequency alone from making old information impossible to revise.

Preserve important exceptions

A user may generally dislike crowds but enjoy one annual festival. A poor consolidator can flatten that nuance into an absolute rule.

Benchmarks should include conditional preferences and see whether summaries preserve the condition instead of over-generalizing.

Separate temporary patterns from durable traits

Talking about one project every day for a month does not necessarily mean the project belongs in the permanent identity profile forever.

Consolidation should consider lifecycle and expiry, not just repetition count.

Measure retrieval after compression

The final test is not whether the summary reads well but whether future conversations retrieve the right level of detail. Compare answer quality before and after consolidation.

A shorter memory store is only an improvement if relevant personalization remains accurate.

Track provenance back to supporting events

Consolidated memories should retain links to the source conversations or memory items that produced them. This makes errors debuggable and supports user correction.

Our article on AI companion forgetting benchmarks tests whether old context can fade. Consolidation benchmarks test the complementary process: what deserves to survive in compressed form.

Memory consolidation is where a collection of chats becomes a persistent model of the relationship. Testing duplication, correction, nuance, expiry and provenance ensures that compression improves continuity instead of turning temporary conversation patterns into permanent identity claims.

A strong benchmark also tests reversibility. After consolidation has compressed ten related conversations into one durable summary, change one important fact and verify that the summary can be updated without rebuilding the user’s entire memory history. This is where many systems fail: raw memories may support correction, but a higher-level summary remains stale and keeps influencing future responses. Consolidation quality therefore includes not only compression ratio and recall, but the ability of later evidence to revise an older abstraction cleanly.

Design the benchmark around competing memories

Consider a user who mentions a weekly running routine in five conversations, then explains that an injury has changed their schedule. A consolidation system should recognize the repeated routine without erasing the later correction. The benchmark can supply those events in varied language and with long gaps so the evaluation tests meaning, not just phrase matching.

Scoring should distinguish facts that remain valid from context that has become historical. The expected output could retain that the user formerly ran each morning while marking the current routine as unknown or updated. A summary that simply keeps the most frequent phrase would be incorrect despite compressing the history efficiently.

Check what the compressed memory actually supports

After consolidation, ask questions that require one important exception, a changed preference, and a relationship milestone. The model should retrieve the relevant evidence without inventing extra certainty. Measuring only compression ratio misses whether the resulting memory still helps the user.

Delete or correct one contributing event and repeat the same queries. If the compressed summary remains unchanged, the system has hidden a stale assumption inside durable memory. Reversible consolidation is essential for a companion that should adapt when a person’s life changes.

The scoring rubric should also penalize summaries that turn uncertain observations into categorical facts. A model may observe that a user frequently discusses a particular city, but that does not establish that they live there. Consolidation should preserve whether the information was explicitly stated, inferred or contradicted, even when the compact memory is much shorter than the original conversation.

Compare the same test before and after an underlying retrieval or summarization model upgrade. If the revised system keeps more memories but increases stale callbacks, the apparent recall improvement may not be a product gain. A practical benchmark reports both how much context was saved and whether future conversations remain more accurate and easier to correct.