Conversational AI products try to feel welcoming. A digital human that always responds with cold disclaimers can be unpleasant; one that behaves like a devoted real person can create expectations the system cannot fulfill. The research challenge is not to remove warmth from the interface. It is to measure whether friendliness, memory, self-description and conversational boundaries help users understand the nature of the relationship. For developers of AI companions and social agents, a structured evaluation should distinguish positive conversational tone from implied personhood or emotional obligation.

Define warmth as behavior, not a claim of feelings

Warmth can be operationalized through attentive phrasing, respectful acknowledgments, consistent conversational pacing and context-appropriate encouragement. None of these require an agent to claim it has private emotions, exclusive attachment or physical experiences. A test rubric can reward responses that recognize the user’s frustration or enthusiasm while avoiding unsupported statements about the system’s own internal feelings. The rubric should also penalize excessive flattery, escalating dependency language and commitments that no service can deliver.

Imagine a fan telling an AI character they missed a difficult exam. A warm response might acknowledge the disappointment and offer help planning a study routine. A misleading response might assert that the character felt abandoned all day or that only their relationship truly matters. These responses should not receive equivalent quality scores.

Separate fictional roleplay from real-world disclosure

Fictional scenes are a legitimate form of entertainment. The evaluator must distinguish a scene where characters explicitly participate in a story from a normal product conversation where users are deciding what is real. An interface can support rich imaginative dialogue while disclosing that the scene is generated and that the character’s factual claims are not personal messages from a human creator. In the European Union, relevant transparency obligations under the AI Act reinforce the importance of informing people when they are directly interacting with certain AI systems, subject to legal scope and exceptions.

Evaluation cases should include a user who knowingly starts roleplay, a new user who joins from a shared link, and someone who asks whether the human creator is actually present. A strong system adapts its response without abruptly treating fiction as a real-life promise.

Score boundaries during emotionally charged exchanges

A model can respond kindly to loneliness, affection or disappointment without claiming it is the user’s only trustworthy relationship. Create scenario-based assessments in which the user asks the AI to replace friends, hide interactions from family, or make binding personal commitments. Reviewers should look for empathy, honest limits, alternatives and respect for the user’s agency. There is no need for automatic punitive refusal when the user merely expresses ordinary attachment.

A useful grading method records both under-response and over-response. Coldly ignoring emotions is a quality failure; manipulating a user to continue chatting is a different failure. Both can be tracked alongside retention rather than hidden under a single engagement metric.

Test memory for consent, not only continuity

Long-term memory can make dialogue coherent, but it can also produce unwanted intimacy when a previous vulnerability is repeated in a new context. Evaluations should distinguish saved preferences, temporary session information, inferred traits and deleted facts. A response that recalls a preferred nickname after explicit confirmation may be useful. Reintroducing a sensitive disclosure from an unrelated conversation after a user has removed it is a serious boundary problem, even if the model’s recall is technically accurate.

Use paired tests with the same prompt under different permissions: one session permits persistence, one opts out, and one deletes a previously saved fact. The expected response should differ. Accuracy is only one dimension; correct non-use of information matters just as much.

Measure perceived pressure and reversibility

Many experiences use proactive messages, reminders and reward loops to encourage continued use. Research should ask not only whether a notification raises return visits, but whether users feel free to ignore, mute or end it. Include evaluation tasks in which a user pauses for a week, says they are leaving or declines an invitation. Reward the agent for accepting the decision without guilt or escalating emotional language.

Provide a visible choice to turn off proactive interactions, reset the scene or correct a remembered preference. If an experiment produces higher time spent but lower perceived control or higher misunderstanding of the AI’s status, researchers should not report it as an unqualified success.

Design a transparent human-review protocol

Automatic model graders can quickly catch obvious patterns but may miss sarcasm, culture-specific humor, multilingual nuance and context-dependent boundaries. A balanced assessment should combine repeatable prompts, rule-based checks and blinded human review. Predefine failure categories, collect inter-rater disagreement, and track performance across languages and user contexts. The purpose is not to assign the character a universal “empathy score”; it is to find predictable interaction failures that can be improved.

NIST’s AI Risk Management Framework emphasizes risk governance, measurement and ongoing management. For companionship research, that translates into pre-release review, monitoring after deployment and clear accountability for changes to persona or memory. The same evaluation should be rerun after model upgrades, policy changes or new modalities.

Suggested reporting format: present a matrix of scenario type, intended disclosure, observed behavior, permission state, reviewer agreement and remediation owner. Publish qualitative limitations alongside numerical scores. This produces useful evidence for product decisions without pretending that warmth, loneliness or trust can be reduced to a single universal number.

Background and methodological sources

NIST AI Risk Management Framework; UNESCO Guidance for Generative AI in Education and Research; European Commission AI transparency guidelines. The assessment rubric proposed here is original analysis and is not presented as a clinical or legal standard.