A research model for relationship memory that combines text, voice, images and video while separating durable facts, episodic events, emotional cues and privacy-sensitive media.

Memory is becoming multimodal

Companion systems increasingly receive voice, photos and video in addition to text. A memory architecture that stores only transcript facts misses important context, but storing every media file forever creates privacy and cost problems.

Separate semantic facts from source media. A photo may establish that a user visited a place, but future conversations may only need the approved fact, not the original image. Systems can extract a compact memory while applying separate retention rules to source media.

Voice carries interaction cues

Tone and pacing can affect conversation, but emotional inference is uncertain. Durable memory should avoid turning a momentary vocal cue into a permanent psychological label. Short-lived conversational state is safer than long-term inference.

Video adds identity and scene context. Video can contain people, locations and activities. Retrieval should be purpose-specific and permission-aware. The system should not surface old visual details merely because they are semantically similar.

Summarization across modalities

A relationship summary can combine selected facts from text and media while preserving provenance. Each memory should know whether it came from an explicit statement, user-approved media interpretation or model inference.

Deletion must propagate. Removing a photo should also address derived embeddings and summaries where appropriate. Tuikor Research’s relationship memory framework explains why forgetting is part of memory quality.

Evaluation agenda

Measure retrieval precision, correction rate, privacy errors, multimodal consistency and user satisfaction. The research goal is not maximum recall; it is context that improves the relationship without exceeding the user’s expectations.

Use a layered multimodal memory record. A voice note can contain a transcript, speaker attribution, emotional context and an original audio asset. A shared image may contain a visual scene, people or objects, an explicit user description and a link to the source file. Keeping these distinct helps the system retrieve meaning without sending every media file back to the model on each turn.

Each memory item should store provenance, creation time, confidence and the consent level applicable to future use. A model inference about a photograph is not equivalent to a fact the user stated about it, and the interface should avoid converting the former into unquestioned personal history.

Keep raw media separate from retrieval summaries

Most ordinary conversation turns need a compact semantic summary rather than the original full-resolution image or video. The media can remain in protected storage and be loaded only when the user asks to revisit it or a specific task requires it. This lowers inference cost and narrows exposure of sensitive material.

Deletion is more complex than removing one visible file. Derived embeddings, captions and relationship summaries may contain information from the removed media. The memory system needs an end-to-end deletion workflow and a way to test whether the information truly stops influencing responses.

Test cross-modal references over time. Evaluation should include a user who shares a photo, later refers to “that place we talked about,” and then changes a related preference. The system should find the appropriate event without confusing generated media with real-world experiences or assuming that all content remains relevant indefinitely.

Useful metrics include retrieval accuracy, source confusion, correction success, latency and cases where a private asset was surfaced unexpectedly. The research goal is selective continuity with strong user control, not unlimited storage.

Multimodal memory should not quietly infer facts about people’s relationships from facial expressions or background audio. A photograph can provide useful scene context, but a model-generated caption is not equivalent to a user’s explicit confirmation. Keeping source type and confidence visible internally helps prevent accidental profiling.

In addition, media often has a much larger privacy footprint than text. A short transcript may be enough to answer a future question, while retaining the full voice sample may introduce a separate biometric risk. Retention policies can keep compact, consented summaries and retire raw media earlier where appropriate.

Finally, compare retrieval with and without the media layer using the same user task. If the original image is rarely needed to improve a future answer, an expensive permanent multimodal archive may not justify its operational or privacy cost.

For voice and video, the product should also distinguish a user-created keepsake from material temporarily uploaded for analysis. A video sent to ask about one scene should not automatically become an enduring relationship artifact. Making that lifecycle visible gives users control over how new modalities contribute to long-term memory.