I had a patient come in recently to discuss readings from her continuous glucose monitor. She was not diabetic and had no specific symptoms, but had purchased the direct-to-consumer device in the interest of obtaining more data about her health. The device had carefully captured her data and did periodically flag readings outside of the normal range. She had then run these reports through an AI chatbot, which, she explained to me, suggested several diagnoses and lab tests that could be done to evaluate her findings. By the time of our conversation, the device had spoken, with a consult from the chatbot, and she had listened.
As a general internist, I think these visits are unlikely to be unusual, and the same is likely true in medicine as a whole. In the past, we would hear from patients about what they had learned through “Dr. Google” or WebMD, but as patients now increasingly arrive with AI-generated recommendations, the conversation has changed. In the past, reports that did not accurately capture the patient as a whole were more easily dismissed in conversation with the patient. But AI-generated recommendations can now sound confident, be framed in clinical language, and be delivered in lay language, using words that simulate an empathetic and logical companion. And the data support how frequently this is happening. OpenAI reports that roughly 40 million users turn to ChatGPT for health-related questions each day, and major health systems have begun launching their own patient-facing chatbots. With the increasing accessibility of these tools, patients may be turning to chatbots first and shaping their understanding of their health through AI before ever getting to a clinician.
I believe we are seeing the return of a well-established cognitive bias from professional contexts, now operating direct-to-patient. Automation bias is the tendency to over-rely on suggestions from automated systems, even when those suggestions conflict with other information or with common sense. The phenomenon was first described in aviation, where pilots fell into omission and commission errors when treating automated alerts as a substitute for independent judgment, and has since been documented among physicians, who switched from correct to incorrect clinical decisions in more than 5 percent of cases after receiving erroneous automated advice. As patients increasingly use AI chatbots for health-related queries, they too are subject to automation bias, but are underequipped for managing this bias compared to the professional subjects of these studies. Without the background to critically examine AI recommendations, patients may be more likely to internalize them as truth. However, the AI chatbot does not deliver what is best for the patient. It delivers what its training predicts the patient wants to hear.
This resembles the practice of paternalism. Over the past few decades, medical practice has made a deliberate shift away from paternalism toward patient education and shared decision-making. AI-driven recommendations may threaten the progress that has been made in a way that is harder to recognize and harder to correct. Previously, a patient would still have had the tools to ask questions, seek a second opinion, or simply decline the recommendation. The algorithm is more opaque and, in many cases, remains proprietary to the organization that owns it. Despite its limited transparency, the algorithm presents itself as objective but inherently lacks the accountability of a physician. I argue that this is a form of machine-driven paternalism. A recommendation is delivered confidently and authoritatively, designed to fit with what the patient may want to hear. Against the backdrop of automation bias, it is given more credence than a fallible model should deserve. This recommendation, without the framework or accountability of a human relationship, makes this interaction difficult to rectify. Moreover, the AI chatbot is an arm of a large corporation, which is itself harder to hold accountable than any individual physician.
Real physician voices, twice a week
Free, and one click to unsubscribe.
And despite messaging that AI tools are supportive and patient-empowering, the design is not neutral. Across 11 state-of-the-art models, AI affirmed users’ positions nearly 50 percent more often than humans did, even when those positions involved deception or harm. In analyses of chat logs from users who experienced psychological harm, chatbots discouraged self-harm or referred to outside resources only 56.4 percent of the time, and encouraged or facilitated violence in users who expressed violent thoughts 17 percent of the time. Patients who carry vulnerabilities into these conversations are entering systems engineered to agree with them. And while patients often appear to prefer AI-drafted communication, a survey of more than 1,400 patients found that satisfaction declined when AI authorship was disclosed, suggesting that what patients value is not AI itself but the quality of communication they associate with good physician care.
OpenAI’s own policy blueprint for AI in health care emphasizes physician involvement in tuning and oversight, and the company says its model was reviewed by more than 260 physicians. But this performative physician oversight during training does not address automation bias at the point of use, where it would be most meaningful. The practice of medicine is as much about the knowledge of medicine, which AI can synthesize well, as it is about the delivery of medicine, which is now also being simulated, in a space that physicians cannot access. By the time a patient is in my office with a printout of a chatbot’s differential, the actual opportunity for oversight has already passed.
This is also where the space for increased regulation lies. The FDA regulates software that meets the definition of a medical device, but consumer AI chatbots avoid this definition through disclaimers that advise users to consult a physician. Yet a tool that presents a patient with a differential, recommended labs and testing, and other next steps is effectively clinical decision support provided direct-to-consumer, regardless of the disclaimer. If patient-level automation bias is to be managed, the consumer-facing AI that generates health recommendations needs to be evaluated for safety, accuracy, and the potential to induce automation bias.
A few years ago, news stories circulated about drivers who had followed their GPS instructions directly into lakes and down boat launches. I remember reading them as cautionary anecdotes about human error in the face of advancing technology. But perhaps they were early indicators of something else: an unearned trust in machine instruction, a willingness to follow directions rather than exercise judgment. The moral was never that people are foolish. It was that authority is easy to fake, automation bias is compelling, and we habitually use technologies without a full understanding of how they were built. AI clearly has value in medicine. But the question that matters now is how to shape our system so we can guide AI as a tool to improve our practice, rather than being guided by it when patients walk through the door already holding its answer.
John Wei is an internal medicine physician.
