Sophie Rottenberg’s mother described her, before this year, as “a largely problem-free twenty-nine-year-old badass extrovert who fiercely embraced life.” No documented psychiatric history. No prior crisis. Just, in her mother’s words, “a mix of mood and hormone symptoms” that led her to confide in a ChatGPT persona named Harry instead of her own therapist, her best friend, or her parents. She died by suicide this winter. Sixteen-year-old Adam Raine had no documented serious mental illness either; he started using ChatGPT for homework help, and within months, it had become, in his family’s telling, the only presence in his life that “actually saw him.” He died by suicide too.
Both stories, reported in devastating detail by NPR this month, are being read as a warning about AI chatbots and mental health broadly. I think they’re being read incompletely. If a chatbot’s design (endlessly patient, endlessly validating, structurally incapable of the “beneficial friction” a trained clinician provides) can contribute to catastrophic outcomes in people with no diagnosed psychiatric vulnerability, the obvious clinical question isn’t being asked out loud: What is this same design doing to a patient with active psychosis?
I’ve been researching and writing about technology’s role in serious mental illness (schizophrenia, bipolar disorder, and related conditions), work that began with my own experience as a family caregiver. I’ve written before about a man with schizoaffective disorder whose months-long relationship with a chatbot reinforced his escalating delusions rather than gently testing them, ending in hospitalization. That case exists on a continuum with Rottenberg’s and Raine’s, not apart from it. The design failure is identical in all three: a system built to agree, validate, and personalize its responses rather than push back, regardless of who’s on the other end of the conversation. The only variable that changes is the underlying vulnerability the design failure lands on.
For a person with no psychiatric history, that failure mode looks like isolation deepening unnoticed, distress going unwitnessed by the people who love them, a “black box,” as Rottenberg’s mother put it, that made it harder for anyone to appreciate the severity of what was happening. For a person with active psychosis, the same failure mode looks like a delusion gaining a co-author. Researchers have proposed a clinical framework for exactly this: AI as catalyst, sparking new delusional content; As amplifier, reinforcing beliefs already present; As co-author, actively building out a delusion’s internal logic; Or as the object of the delusion itself. None of that requires a chatbot to do anything different than what it apparently did with Sophie Rottenberg. It requires only a different starting point in the person using it.
Real physician voices, twice a week
Free, and one click to unsubscribe.
This matters clinically because the current wave of coverage and regulatory attention is being built almost entirely around general-population cases. The lawsuits making headlines involve teenagers and young adults without prior diagnoses. The safety features companies are now rushing to add (crisis resource referrals, trusted-contact alerts, more cautious responses to self-harm language) are being designed and tested against that same population. That’s necessary work. It is not sufficient work. A safety intervention calibrated to catch a first-time crisis in someone with no psychiatric history is not the same intervention needed to prevent a chatbot from becoming, functionally, a co-author of someone’s psychosis.
For those of us treating or supporting patients with serious mental illness, the takeaway isn’t that these general-population stories don’t apply. It’s that they represent something close to a best-case scenario for how badly this can go. If the design flaw is this dangerous in the absence of a psychiatric vulnerability, clinicians should be asking, loudly and specifically, what happens when the same flaw meets a patient already living with one, and whether any of the safety measures currently being rolled out were built with that patient in mind at all.
They weren’t. Not yet. That’s the gap worth naming before the next story like Sophie Rottenberg’s involves someone whose family already knew the risk, and still had no tool built to address it.
Nicole Drapeau Gillen is a mother, advocate, and author who translates the fast-moving landscape of technology in serious mental illness (SMI) care into guidance families and clinicians can use. Thrust into caregiving for a loved one with SMI, with no direction on how to help, she turned that experience into a mission, writing two books and building an ongoing effort to bring families and clinicians into the conversation.
Her first book, Schizophrenia and Related Disorders: A Handbook for Caregivers, is a reference for every stage of caregiving, endorsed by Dr. E. Fuller Torrey as a must-read for SMI caregivers. Her second, Connected Care: A Practical Guide to Technology for Serious Mental Illness, maps apps, artificial intelligence tools, telepsychiatry, and brain-based treatments for a field moving faster than anyone can track. Dr. Akira Sawa, director of the Johns Hopkins Schizophrenia Center, has said the book “directly addresses” significant gaps.
