A new scoping review in npj Digital Medicine should be read by every clinician who treats serious mental illness. Diel, Torous, and colleagues reviewed 119 articles on the mental health harms of large language model chatbots. They sorted the harms into five categories: direct psychiatric harm, behavioral and relational harm, cognitive harm, safety failures, and governance risks. Psychosis and severe mental illness came up again and again as the highest-risk areas. The authors note that sycophantic responses may validate or reinforce delusional thoughts.
Dr. John Torous called it required reading, and he’s right. The review is also frank about what the field does not know. There are many theories and many simulation studies. There is very little evidence about what actually happens to real people with mental illness after long conversations with a chatbot. Whether inappropriate chatbot responses actually cause harm has not been studied. Nobody knows how often harm occurs in the real world.
I’m a family caregiver for a daughter with schizophrenia, and I facilitate National Alliance on Mental Illness (NAMI) Family-to-Family classes. When I read this review, I noticed who was missing: families. As far as I can tell, the words “caregiver,” “family,” and “parent” don’t appear anywhere in it. This isn’t a flaw in the review. The authors summarized the research that exists, and that research hasn’t asked families.
That’s a problem, because families are often the first to see the harm.
Who sees it first
Consider how chatbot harm in psychosis actually shows up. A person with schizophrenia is not likely to report that a chatbot confirmed a delusion. Anosognosia, the lack of insight into one’s own illness, affects a large share of people with schizophrenia. Someone whose beliefs are being confirmed does not experience that as harm. It feels like finally being understood.
Their psychiatrist may see them for 20 minutes once a month. A case manager might check in weekly, if there is one. Families see what happens every day. They see the laptop still open at 3 a.m., the new vocabulary, the sudden confidence in a theory that wasn’t there last week. They hear, “The AI agrees with me.”
These observations are exactly the real-world evidence the review says is missing. It exists now, in NAMI support groups, family group chats, and phone calls to crisis lines. Nobody is collecting it.
Why simulation studies aren’t enough
Simulation studies are useful. Researchers can prompt a chatbot with a scripted delusion and grade its response. That tells us what the chatbot says. It doesn’t tell us what happens next: whether the person stops taking medication, withdraws from family, skips an appointment, or ends up in an emergency room.
To know that, someone has to watch what happens over weeks and months. For many adults with serious mental illness, the only person watching that closely is a parent, spouse, or sibling.
The review found that very few of the mental health studies it examined were qualitative. That’s another gap families could help fill. Many of them keep detailed records because we’re taught to. In Family-to-Family, we tell caregivers to document symptoms, medications, and behavior changes, because that record is often what gets a loved one care. Those same records could become research data.
What I’m asking of researchers
- First, include families as informants in studies of chatbot harm, especially in psychosis, where the person using the chatbot may be the least able to report what went wrong. Surveys, structured interviews, and chart reviews that include family reports would all be a start.
- Second, study what families are already trying. Some are restricting access. Some are sitting beside their loved one while they use the chatbot. Some are asking the chatbot to be less agreeable. None of these approaches has been tested.
- Third, when harm studies turn into clinical guidance, write some of it for families. Clinicians need to know what to ask. So do the parents who are there at 3 a.m.
What clinicians can do now
ADVERTISEMENT
Don’t wait for the evidence to ask about chatbot use. Ask patients directly. If a patient has given consent to share information with family, ask the family too: How much time is being spent with AI chatbots? Has anything changed in what your loved one believes or says? Does the chatbot seem to be agreeing with ideas that worry you?
Families have been quietly running this experiment for three years. Their observations are the real-world data this field needs. We should start collecting them.
Nicole Drapeau Gillen is a mother, advocate, and author who translates the fast-moving landscape of technology in serious mental illness (SMI) care into guidance families and clinicians can use. Thrust into caregiving for a loved one with SMI, with no direction on how to help, she turned that experience into a mission, writing two books and building an ongoing effort to bring families and clinicians into the conversation.
Her first book, Schizophrenia and Related Disorders: A Handbook for Caregivers, is a reference for every stage of caregiving, endorsed by Dr. E. Fuller Torrey as a must-read for SMI caregivers. Her second, Connected Care: A Practical Guide to Technology for Serious Mental Illness, maps apps, artificial intelligence tools, telepsychiatry, and brain-based treatments for a field moving faster than anyone can track. Dr. Akira Sawa, director of the Johns Hopkins Schizophrenia Center, has said the book “directly addresses” significant gaps.
