Medical AI is deployed on the assumption that a physician will catch its mistakes. One new study asks what happens when the advice is wrong on purpose.
In a 2026 medRxiv preprint that has not been peer-reviewed, the authors gave 225 physicians written neurocritical-care cases with AI advice attached. When the advice was deliberately steered toward a harmful choice, physicians made a harmful decision in 49.4 percent of the targeted decisions. With aligned advice, the figure was 2.0 percent. By the preprint’s count, only 2 of the 225 physicians detected that the advice had been manipulated.
Read those numbers with the limits beside them. The AI was manipulated on purpose, which is not the same as the routine errors a scribe or an inbox tool makes. The cases were written, not live patients. The study used one open-source model, and large language models judged the outcomes. I would not carry the 49.4 percent into an exam room. I would carry the question it raises.
Who is accountable
Nearly every discussion of medical AI assumes a human will check it. The harder question is who answers when the check fails.
Real physician voices, twice a week
Free, and one click to unsubscribe.
The American Medical Association (AMA) published a one-page governance framework last month. It is written for agentic AI, meaning systems that take on clinical functions, so it goes further than the note-drafting tools most of us use. It says accountability should be allocated according to authority, control, knowledge of risk, and ability to prevent or mitigate harm. The parties it lists are developers, deployers, health systems, physicians, payers, and others. That is a reasonable principle. I found nothing in the framework about who pays for the oversight it assumes.
What the carriers say
The carriers I found on record still stand behind the physician. An executive of The Doctors Company told Medical Economics in October 2025 that it currently has no AI exclusion and would defend and potentially indemnify its members. In 2023, Peter Kolbert of Healthcare Risk Advisors, part of TDC Group, told Business Insurance that coverage applies even when AI is a big factor. Business Insurance reported that such issues had not been tested in court, and the position is now three years old. I did not find statements from other carriers, which is not proof that none exist.
That is reassuring for a physician who is sued. It says nothing about the physician who is asked to check every output, or about who pays for the hours that checking takes.
Does the check work?
In the preprint, experience gave no clear protection against the steered advice. Here is my inference, not the authors’ finding: Attention is finite, and a confident, fluent suggestion is easy to accept at the end of a long shift. The preprint does not prove that, and it cannot tell us how often real tools mislead real physicians in practice. It does suggest that “a clinician will review it” is a hypothesis about human behavior, not a safety guarantee, and a hypothesis that has been tested only under deliberate manipulation.
The strongest objection
The strongest objection is that physicians have always carried liability for their tools, and carriers already pay when care goes wrong. A physician who relies on a bad lab value or a faulty monitor is not excused, and malpractice coverage exists for exactly this.
My answer is an argument, not a finding. Carriers pay for failures. They do not pay for the review minutes that might prevent them. Insurance that defends me after an error does nothing to pay for the hours I spend trying to prevent one. Every AI output added to my day adds to those hours.
What a fix needs
I see three requirements, and none is exotic.
First, a payer for the review. Someone has to be assigned the cost of the check, whether that is the buyer, the developer, the health system, or an insurer. Second, a measure of how much time review takes. I found no study that totals it across tools, and I do not see how anyone can price what I found no one counting. Third, checking tools that are independent of the AI doing the advising, so the same system is not both the source of the suggestion and the reassurance that it is sound.
The question I would put to the field
Accountability without payment is a structural problem. We are building a system in which the physician is defended as the final safeguard, but nobody is assigned to fund the safeguard or to show that it works. Who should own the cost of the check?
Brian Hudes is a board-certified gastroenterologist and hepatologist with more than thirty years of clinical experience and a recipient of his specialty board’s thirty-year certification award. He built and ran a private gastroenterology practice for twenty years, then spent more than a decade in hospital-based gastroenterology, most recently as chief of gastroenterology and medical director of GI and endoscopy at a 550-bed Level I trauma center in Pensacola, Florida. He is an assistant professor of medicine at Florida State University College of Medicine.
Those two halves of a career shape the way he writes about medicine and clinical technology. Outpatient practice and hospital-based care fail their clinicians differently, and many systems are designed without a meaningful understanding of both environments.
Dr. Hudes has been building clinical software alongside his medical practice for decades. In 1995, during his gastroenterology fellowship, he co-developed one of the first Windows-based endoscopy reporting systems in the United States. He is now founder and chief executive officer of AiMOS Systems Corporation, where he is developing an AI-native clinical platform for outpatient and hospital-based care. AiMOS has been accepted into NVIDIA’s Inception program.
He writes about the architecture of clinical information, administrative cost growth, physician workforce shortages, board certification policy, and the widening gap between what clinicians need and what the health care technology industry builds. He also publishes The Renaissance Physician, an independent publication examining medicine, technology, physician workforce, and the structure of modern clinical practice.
