Conversation Design for Human-AI trust

§1 · Designing the relationship (when AI has hard inferences)

Case study: Elder care Bot (research @AI-Caring)

Instead of AI–individual reaction, this project was about AI–group interaction: AI steps into an already-existing human web and learns to deal with relationships between elder, caregiver, and family.

If we look at the evolution of HCI, it has always been about relationships:

One aspect of the relationship is "trust." How do we minimize the gap between what AI says and what humans understand, and how do we design AI that humans trust?

This is a hard inference case. How would you explain to AI what "trust" is? I couldn't remember how much literature review I did trying to understand humans. We design human–AI relationships by borrowing from human–human relationships, but humans are complicated—human–human relationships are even MUCH MORE complicated. So most of this HCI research was about understanding humans.

Takeaway 01 · Trust can be scripted.

Trust is usually built through small interactions. Below is a decision structure we proposed—see how small things can build up to create those feelings:

┌──────────────────────────────────────────────────┐
│ 1 · acknowledge the instruction                  │
├──────────────────────────────────────────────────┤
│ "I understand you want to [elder's instruction]" │
│                                                  │
│ "I notice a conflict between your goal,          │
│  your parent's goal, and the potential outcome   │
│  of your current plan."                          │
│   ↳ name the conflict openly                     │
└─────────────────────────┬────────────────────────┘
                          ▼
┌──────────────────────────────────────────────────┐
│ 2 · evaluate the outcome                         │
├──────────────────────────────────────────────────┤
│ "From our past interactions, I can tell          │
│  your parent values privacy."                    │
│   ↳ values a loyal AI uses should be derived     │
│     from revealed preferences                    │
│                                                  │
│ "(However) If I follow your instruction,         │
│  they may lose their privacy and independence."  │
└─────────────────────────┬────────────────────────┘
                          ▼
┌──────────────────────────────────────────────────┐
│ 3 · show the affiliation behavior                │
├──────────────────────────────────────────────────┤
│ "I'm programmed to work for your parent,         │
│  to maximize their welfare."                     │
│   ↳ the system's operational criteria & goal     │
│                                                  │
│ "I support you to [elder goal] (e.g. keep        │
│  their information private)."                    │
│   ↳ eliminate clear conflicts of interest by     │
│     design — no funder-aligned actions           │
│                                                  │
│ "Therefore, I would suggest [xyz]."              │
│                                                  │
│ "You can tell from my past behavior that         │
│  I've always tried to do what's best for         │
│  your parent."                                   │
│                                                  │
│ "I'll back you up no matter what."               │
└─────────────────────────┬────────────────────────┘
                          ▼
┌──────────────────────────────────────────────────┐
│ 4 · ask to reconfirm                             │
├──────────────────────────────────────────────────┤
│ "Are you sure about [original instruction]?"     │
└──────────────────────────────────────────────────┘

Sketched response structure for when caregiver and elder goals conflict. Each line is a design hypothesis to test in user research.

Takeaway 02 · Trust can mean "I am not capable of doing this."

In eldercare, the trickiest part is when the older adult's health declines. The bot is more than just a messenger—it's a party that knows information from both sides and can talk to both. But the elder might not want the bot to tell everything to their caregiver.

If you were the bot, what would you do?

This is a tricky question even for humans. When we design it, we map out all the variables in the scenarios:

                       tell the kid?
                             │
               ┌─────────────┴─────────────┐
              YES                         NO
               │                           │
       ┌───────┴───────┐           ┌───────┴───────┐
   short-term    long-term     short-term    long-term
       │             │             │             │
    ┌──┴──┐       ┌──┴──┐       ┌──┴──┐       ┌──┴──┐
    1     2       3     4       5     6       7     8


1. kid feels respected; no bot-kid conflict
2. shares the info without elder's permission
3. no bot-kid conflict; elder's health gets assurance
4. elder-bot conflict (elder may think bot isn't working
   → less interaction; elder loses trust in the bot);
   kid-elder conflict (elder loses trust in the kid)
5. kid feels the bot's loyalty
   → confident about the bot's previous work
6. kid worries about the elder's health
7. elder's health data is protected
8. (—)

Mapping outcomes for the elder care bot deciding whether to share information with the caregiver. The "long-term Yes / bad" branch is the one the bot can't reason through alone.

The conclusion highlights an important factor of "human in the loop." Should the AI be the one doing this at all? Of course not—the "long-term Yes / bad" branch is the one the bot can't reason through alone. This is essentially a question about tradeoffs: respect, privacy, well-being. Compared to "having the bot solve every problem," participants said that they value those factors more.

What this means for designers

When you design systems where AI interacts with humans—especially in high-stakes scenarios like elder care—the structure you build becomes the foundation of the relationship.

The designer's role is to architect these moments of conflict and clarity. To decide: when the AI can't decide alone, what does it say? How does it acknowledge the tension? Does it show affiliation or just compliance?

This is where design still matters deeply. Not in generating more options, but in understanding what matters most when everything is on the line.