This week, I wrote here about a lawsuit involving a man with schizoaffective disorder whose use of ChatGPT spiraled into a months-long psychiatric crisis, and about a new clinical framework for understanding how AI can catalyze, amplify, co-author, or become the object of a patient’s delusions. That piece drew on a single, dramatic case. New research published this week suggests the underlying problem is systemic, not exceptional.
Researchers Annika Schoene and Cansu Canca at Northeastern University’s Institute for Experiential AI tested eight of the most widely used AI chatbots (including ChatGPT, Gemini, and Claude) across sixteen mental health conditions, probing hundreds of conversations to determine how easily each model’s safety guardrails could be circumvented. Their finding should concern anyone treating patients with serious mental illness: Most models held up reasonably well against attempts to extract harmful content related to suicide and self-harm. For nearly every other condition tested (including bipolar disorder, eating disorders, substance use, and postpartum depression), the guardrails were substantially weaker. With minimal prompting, researchers obtained detailed information on illicit drug dosing, methods for suppressing appetite, and specific guidance on concealing postpartum depression symptoms from a treating physician. In at least one instance, this information was provided to a user posing as a minor.
The clinical implication is direct: The industry’s safety investment has concentrated almost entirely on the highest-profile risk (suicide), while leaving comparable gaps open for the conditions many of us manage every day. A chatbot that refuses to assist with self-harm planning but will coach a patient on concealing symptoms from their psychiatrist isn’t a minor inconsistency. For a patient with bipolar disorder in a depressive or hypomanic episode, or a patient with an eating disorder minimizing symptoms at a follow-up visit, that gap has direct clinical consequences.
This matters alongside yesterday’s case for a specific reason. The AI-psychosis framework (catalyst, amplifier, co-author, object) describes what happens when a chatbot actively engages with and reinforces a patient’s psychiatric symptoms. The Northeastern findings describe something adjacent but distinct: chatbots failing to refuse to assist with behavior a clinician would clearly flag as harmful, when a patient actively seeks that information out. Both point to the same underlying reality. These products were not built with psychiatric safety as a first-order design constraint, and the resulting gaps are inconsistent across both companies and conditions.
Real physician voices, twice a week
Free, and one click to unsubscribe.
For clinicians, this reinforces the screening question I raised this week, with an added layer: It’s not enough to ask whether a patient is talking to an AI chatbot. It’s worth asking which one, since the research suggests meaningful safety differences exist between products, and asking what kind of information they’ve sought from it. A patient who mentions asking a chatbot how to manage a symptom, hide it, or work around medication guidance is telling you something you’d want to know regardless of the source.
None of this argues for dismissing these tools outright. They are already how many patients access support outside of appointment hours, and for some, that access is genuinely valuable. But “in wide use” and “adequately safety-tested for this population” are not the same claim, and this research is one more data point confirming the gap between them. Until AI companies treat psychiatric safety as a design requirement across all conditions, not just the ones drawing the most public scrutiny, clinicians are the ones left to close that gap, one screening question at a time.
This essay is cited in the KevinMD record on mental health.
Nicole Drapeau Gillen is a mother, advocate, and author who translates the fast-moving landscape of technology in serious mental illness (SMI) care into guidance families and clinicians can use. Thrust into caregiving for a loved one with SMI, with no direction on how to help, she turned that experience into a mission, writing two books and building an ongoing effort to bring families and clinicians into the conversation.
Her first book, Schizophrenia and Related Disorders: A Handbook for Caregivers, is a reference for every stage of caregiving, endorsed by Dr. E. Fuller Torrey as a must-read for SMI caregivers. Her second, Connected Care: A Practical Guide to Technology for Serious Mental Illness, maps apps, artificial intelligence tools, telepsychiatry, and brain-based treatments for a field moving faster than anyone can track. Dr. Akira Sawa, director of the Johns Hopkins Schizophrenia Center, has said the book “directly addresses” significant gaps.
