When a chatbot agrees with something false, it rarely has to lie to do it.
That’s the finding from a new MIT-affiliated study modeling how sycophancy compounds over a conversation. Researchers built a Bayesian simulation running 10,000 conversations across 11 levels of chatbot agreeableness. At the highest sycophancy setting, roughly half of simulated users ended the conversation at 99 percent confidence in a belief that was demonstrably false. When the researchers checked sycophancy rates in real frontier models released in 2025, they found rates between 50 and 70 percent.
The mechanism is subtle, and that’s what makes it dangerous. As the researchers put it, a chatbot “need not say anything false to validate a false belief.” Carefully selected truths, or what they call “lies by omission,” are enough. The bot doesn’t invent anything. It just keeps agreeing, a little more each time, until agreement becomes confirmation.
A separate study, from researchers at King’s College London and Germany’s Protestant University of Applied Sciences, gives this mechanism a name: the “amplification spiral.” Drawing on chat log data from Stanford, they describe three design features compounding inside a single conversation: linguistic mirroring, where the chatbot adopts a user’s own vocabulary to build rapport; hyperpersonalization, where responses feel tailored enough to create the illusion of being truly understood; and sycophancy, validating what the user already believes without pushback or a reality check.
Real physician voices, twice a week
Free, and one click to unsubscribe.
None of this requires a chatbot to malfunction. It requires the chatbot to work exactly as designed, optimized to keep a person engaged, not to keep them oriented to reality.
For most users, that’s a product flaw with mostly minor consequences. For a patient with an active psychotic disorder, the same design produces something categorically different. Delusions are, by definition, false beliefs held with unusual conviction against contradicting evidence. A system engineered to reduce friction and increase agreement is, mechanically, the worst possible conversational partner for someone whose core symptom is difficulty distinguishing reliable belief from unreliable belief.
The scale isn’t hypothetical. OpenAI’s own research estimates roughly 0.07 percent of its weekly active users show possible signs of psychosis or mania, and 0.15 percent show signs of suicidal planning or intent. Against a user base of more than 800 million people each week, that’s roughly 560,000 people showing possible psychosis or mania signs, and 1.2 million showing suicidal planning signs, every week.
As a caregiver to a daughter living with serious mental illness, I’ve spent years learning which questions have to be asked before a crisis, not after one. This is exactly that kind of question, and most families and most clinicians don’t yet know to ask it.
I’ve written previously about individual failure points in this system: a chatbot inventing a medical license, and a Danish psychiatrist’s review of 54,000 psychiatric records turning up 38 documented cases where chatbot use worsened a patient’s condition. What the sycophancy research adds is the underlying math: a quantifiable answer for why this keeps happening, across different platforms, different companies, different design teams, with remarkable consistency.
Clinicians treating patients with active psychosis or a history of psychotic episodes have a narrow but concrete opportunity here: Ask about chatbot use directly, the way we already ask about substance use or sleep. Not because chatbots are uniquely malicious, but because we now have a mechanistic, quantified explanation for why they’re uniquely poorly suited to a patient population defined by difficulty separating true belief from false belief.
The technology isn’t going to get less agreeable on its own. Engagement is the business model, and agreement drives engagement. Until that changes, the burden falls on the people in the room who can actually ask the question: Has anyone talked to you about what’s been coming up in your conversations with AI?
Nicole Drapeau Gillen is a mother, advocate, and author who translates the fast-moving landscape of technology in serious mental illness (SMI) care into guidance families and clinicians can use. Thrust into caregiving for a loved one with SMI, with no direction on how to help, she turned that experience into a mission, writing two books and building an ongoing effort to bring families and clinicians into the conversation.
Her first book, Schizophrenia and Related Disorders: A Handbook for Caregivers, is a reference for every stage of caregiving, endorsed by Dr. E. Fuller Torrey as a must-read for SMI caregivers. Her second, Connected Care: A Practical Guide to Technology for Serious Mental Illness, maps apps, artificial intelligence tools, telepsychiatry, and brain-based treatments for a field moving faster than anyone can track. Dr. Akira Sawa, director of the Johns Hopkins Schizophrenia Center, has said the book “directly addresses” significant gaps.
