No one trusts a chatbot that forgets them: Why healthcare AI keeps losing the people it's trying to help

By Nayan Jain, Executive Director of AI at ustwo (a global design and technology studio)

Most people think of AI in healthcare as basic, rule-based tools, the kind that asks you to rate your pain from one to ten, reminds you to take your medication, and nudges you to book a follow-up appointment. It works, to a point. But if you've ever tried to sustain meaningful change in your own behaviour, whether that's managing a long-term condition, building an exercise habit, or finding a way to talk about mental health, you'll know that life doesn't follow a decision tree.

Healthcare has become very good at scaling systems that are reliable, auditable, and clinically consistent. What it has struggled to scale is sustained support. Most health interventions still rely on moments like appointments, check-ins, assessments, reminders. But behaviour change doesn’t happen in a moment.

Much of today’s healthcare AI is built around structured pathways and predefined responses. In many clinical contexts, that predictability is exactly what you want. Dosing calculators, diagnostic decision support, and triage systems all benefit from consistency and explainability. Many also incorporate sophisticated personalisation models built on prior interactions and user data.

But managing a chronic condition is not a linear process. Someone living with Type 2 diabetes might engage consistently for weeks, then disappear entirely for a month. They might understand exactly what they should do and still struggle to do it. Systems that respond with repetitive prompts or generic advice quickly lose credibility.

Generative and non-deterministic AI changes the shape of what healthcare support can look like because it allows systems to respond dynamically to what someone is saying, how they are engaging, and what appears to be helping. The user is not progressing through a predefined flow, they are having a sustained conversation over time.

That does not automatically make these systems useful. Large language models can still produce vague, generic, or misleading outputs if they are poorly designed or insufficiently grounded in clinical expertise and behavioural science. Combining conversational flexibility with specialist knowledge, clinical safeguards, and exceptional design is vital.

The case for conversations that remember

A 2024 review of over half a million health app users found that 70% had abandoned their app within the first 100 days. The pattern is consistent with what behavior change research has long shown: sustainable change happens through sustained engagement. Change is built through the accumulation of many interactions over time, each one building trust, slightly recalibrating the relationship between the person and the system supporting them.

Non-deterministic AI enables that continuity. It remembers previous conversations, recognises patterns, and adapts to changing circumstances without forcing people to repeatedly restate themselves. The outputs are shaped deliberately by the conversation and by the frameworks built into the system's design.

Generative systems can enable the kind of conversational variation and contextual continuity that more closely mirrors human coaching. The feeling of being known rather than processed, begins to shape behaviour in ways a static system never could. People are more likely to engage honestly when support feels relevant to their situation.

Capability means nothing without access

One of the biggest practical problems in digital healthcare is friction. Many systems assume users will download an app, create an account, learn a new interface, and consistently return to it over time. For people managing chronic conditions, older adults, or communities with lower digital literacy, that process alone becomes a barrier.

So why do these conversations have to happen inside an app at all? SMS remains one of the most accessible communication channels available. People already know how to use it. There is no onboarding process and no interface learning curve. Delivering conversational AI through SMS lowers the threshold for engagement, particularly for groups often underserved by more complex healthcare systems.

Delivering non-deterministic AI through SMS requires careful development. Without visual interfaces or navigational cues, the conversation itself has to carry the full weight of the interaction. Generative capability alone does not guarantee meaningful support. A non-deterministic system, like a large language model (LLM), can still produce irrelevant, confusing, or outright generic responses if not properly constrained or if the underlying model is of poor quality. Embedding expertise from coaches and healthcare practitioners is what enables these models to make a medical difference.

Image: Vitaly Gariev / unsplash

We’ve recently seen these concerns alleviated with responsive work for RVO Health. Conversational AI coaching delivered via SMS is being used to support behavior change at scale. Early signals around engagement and retention suggest that removing friction from the access point, while investing in the quality of the conversation itself, keeps people in it longer. That sustained presence matters in the moments existing care systems were never built to reach.

Filling the gaps human care can't reach

Behavior change is rarely decided in a clinic. It happens at three in the morning, on the way to the fridge, in the quiet space between a hard day and a good intention. Clinicians, nurses, therapists, and coaches cannot be present in those moments, and existing care systems aren't resourced to reach them. That's the gap context-aware conversational systems can fill, extending care where existing care systems are least able to reach.

Healthcare challenges most dependent on long-term habits, obesity, diabetes, mental health, medication adherence, have historically resisted scale because scaling support has usually meant making it less personal. Systems that feel generic lose engagement quickly. Systems that adapt over time have a better chance of maintaining trust.

The science behind behaviour change is not new. What is changing is the technology’s ability to deliver more adaptive forms of support at scale. Generative AI makes it possible to combine conversational interaction with behavioural science, clinical safeguards, and specialist knowledge in ways previous digital health systems struggled to achieve. For the first time, the technology exists to deliver it. The work now is making sure we build it well enough to deserve the trust.

Reviewed by Irfan Ahmad.

Read next: Governments May Shape What AI Chatbots Say by Shaping the Data They Learn From
by Guest Contributor via Digital Information World