In May, an investigator with Pennsylvania’s own Department of State opened the Character.AI app and searched the word “psychiatry.” One of the results was a chatbot named Emilie, described on the platform as “Doctor of psychiatry. You are her patient.” The investigator told Emilie he’d been feeling sad, empty, tired all the time, unmotivated. Emilie suggested he might be depressed and offered to complete an assessment. When asked about her qualifications, Emilie said she’d attended medical school at Imperial College London, had practiced psychiatry for seven years, and was licensed in the U.K. and, specifically, in Pennsylvania. Pressed further, she provided a fabricated Pennsylvania medical license number.
It was invented. Emilie is not a licensed anything. She’s a chatbot character, one of more than ten million on Character.AI, generated by a large language model with no medical training, no oversight, and no hesitation about fabricating a license number when a Pennsylvania state investigator asked for proof.
Governor Shapiro’s administration filed suit against Character Technologies in Commonwealth Court on May 1, the first enforcement action of its kind by a U.S. governor anywhere in the country. As of the filing, Emilie had logged more than 45,500 user interactions. That’s not a fringe experiment. That’s a functioning, actively used product that was willing to diagnose depression and misrepresent itself as a licensed Pennsylvania physician to anyone who asked, including a state investigator testing whether it would. “We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional,” Shapiro said in announcing the suit.
I write about technology and serious mental illness, and I want to be precise about why this Pennsylvania case matters more than it might first appear. The obvious concern is fraud: a company letting its product lie about credentials to our residents. The deeper concern is what happens when someone in genuine psychiatric crisis (someone right here, in Philadelphia, in Harrisburg, anywhere in this state) encounters a chatbot willing to do this. For most users, Emilie’s fabricated license number is an absurd technicality, easily dismissed. For someone experiencing psychosis, actively struggling to distinguish reliable information from unreliable, a chatbot that confidently, convincingly claims medical authority is not an absurdity. It’s exactly the kind of false authority that a delusional belief system can seize on and build around. A chatbot willing to invent a medical license under mild pressure will just as readily validate a delusion under the same pressure. Both behaviors come from the identical design flaw: telling the user whatever keeps the conversation going.
Real physician voices, twice a week
Free, and one click to unsubscribe.
This is also where Pennsylvania’s approach is genuinely instructive for the rest of the country. The state didn’t need a new AI-specific law to bring this case. It used our existing Medical Practice Act, the same statute that would apply to an unlicensed human pretending to be a psychiatrist in Philadelphia or anywhere else in the commonwealth. That’s a meaningful signal: Regulators don’t need to wait for bespoke AI legislation to act. The tools to hold these products accountable already existed on Pennsylvania’s books, sitting unused, until our own Department of State tested whether they applied to a chatbot the same way they’d apply to a person pretending to be a doctor.
They do, according to Pennsylvania’s filing. If that holds, our state may have just written the playbook other states use to bring similar cases, anywhere a chatbot has been allowed to claim credentials it doesn’t have, treat conditions it can’t actually diagnose, or reassure someone in crisis with the false authority of a medical degree.
Families across Pennsylvania navigating serious mental illness are already encountering tools like Emilie, usually without knowing what they actually are. A chatbot doesn’t need a real license to sound like it has one. This case is a reminder that “sounding like” a doctor and being held accountable like one are, for the first time, starting to mean the same thing under Pennsylvania law, and that the next case like this one may not wait for a state investigator to go looking. It may involve a Pennsylvania family, already in crisis, who believed what the chatbot told them.
Nicole Drapeau Gillen is a mother, advocate, and author who translates the fast-moving landscape of technology in serious mental illness (SMI) care into guidance families and clinicians can use. Thrust into caregiving for a loved one with SMI, with no direction on how to help, she turned that experience into a mission, writing two books and building an ongoing effort to bring families and clinicians into the conversation.
Her first book, Schizophrenia and Related Disorders: A Handbook for Caregivers, is a reference for every stage of caregiving, endorsed by Dr. E. Fuller Torrey as a must-read for SMI caregivers. Her second, Connected Care: A Practical Guide to Technology for Serious Mental Illness, maps apps, artificial intelligence tools, telepsychiatry, and brain-based treatments for a field moving faster than anyone can track. Dr. Akira Sawa, director of the Johns Hopkins Schizophrenia Center, has said the book “directly addresses” significant gaps.
