Health care’s AI conversation is moving from documentation to decision. The pitch, repeated across the industry, is that we should stop using AI merely to write our notes and start using it to improve the decisions those notes record: to surface risks, close care gaps, and deliver “actionable intelligence” inside the workflow. The ambition is reasonable. But it quietly raises the stakes on a problem the documentation era let us ignore: A clinical AI can produce an answer that is fluent, well-organized, clinically plausible, and wrong, and the better it looks, the less likely anyone is to catch it.
Picture the ordinary case. A clinician asks an assistant to summarize a patient’s history before a visit. It returns a clean, readable summary and misses that a medication was discontinued six months ago, or merges findings from two different encounters, or states an old diagnosis without making its date clear. Nothing about the output looks wrong. That is precisely the danger. In medicine, judging an AI response on whether it sounds correct is not just insufficient; it is the exact failure mode that hurts patients.
This is why best-case accuracy is the wrong thing to measure, and the headlines keep measuring it. A 2026 Lancet study of Google’s Articulate Medical Intelligence Explorer (AMIE), a conversational diagnostic AI, was widely read as “90 percent accurate.” Read the paper more closely: The final diagnosis appeared in AMIE’s differential 90 percent of the time only when the list extended to seven candidates. Its single most likely diagnosis was correct in 56 percent of cases, and the correct diagnosis appeared within the top three in 75 percent. Those are useful results, but they describe very different levels of diagnostic precision.
The safety result deserves the same care. The study reported zero safety-stop interventions across all 98 completed AI interactions, but every conversation was monitored in real time by physician safety supervisors under prespecified criteria. The supervisors did not need to stop a consultation, although they noted one hallucination and added clinical information in five interactions. The important point is that the system was evaluated inside an explicit human-oversight scaffold. That is different from demonstrating that unsupervised use is safe.
Real physician voices, twice a week
Free, and one click to unsubscribe.
Utah’s live prescription-renewal pilot shows the same structure from a different angle. Physicians agreed with the AI’s renewal recommendations 91 percent of the time, and judged 31 percent of its escalations “too cautious” (notably, not reckless). Erring toward escalation is the right direction for a tool that proposes rather than decides. The state’s current public page says the pilot remains in phase 1, where a licensed medical practitioner authorizes every request. Remove that reviewer and you have changed the intervention, not merely streamlined the workflow.
So the question to ask of any clinical AI is not “How good is its best answer?” It is “What does it do with a hard one?” Five tests matter more than the accuracy score:
- Traceability: Can every clinical claim be traced to a specific source in the record, or is it asserting things you cannot check?
- Temporal context: Does it distinguish current information from historical: an active medication from a discontinued one, today’s problem from a resolved one?
- Contradiction handling: When two records disagree, does it surface the conflict, or silently pick one and present a tidy answer?
- Abstention: When the record does not contain the answer, does it flag the gap, or fill it? A system that confidently smooths over missing information is more dangerous than one that admits it does not know.
- Human review: Can the clinician see and correct the output before it influences care, with enough transparency to actually do so?
Of these, abstention is the one I would test first, because it is the hardest to fake and the most revealing. A model tuned to always produce a fluent answer will always produce one, correct or not. A model willing to say, “These two notes disagree” or “This value is missing” is one built around the consequences of being wrong rather than the polish of being right.
None of these are prompt-engineering tweaks. They are product and architecture decisions (retrieval, source attribution, temporal modeling, validation, and a real human-review step), and the right combination depends on the use case and the cost of an error. That is the uncomfortable implication of the move from documentation to decision: The higher the stakes of the output, the more the design has to be organized around its failure modes, not its demos.
When six of the largest physician organizations warned in September against treating AI as inherently better informed than physicians, the headline was about respect for clinical expertise. The operational reading is more demanding. Keeping the physician central is not a slogan; it is a set of engineering requirements (traceability, temporal context, contradiction handling, abstention, and review) that a fluent summary with no sources and no sense of what it does not know will fail every time. A clinical AI earns trust not by being impressive on its best day, but by being honest on its worst.
A longer companion essay, “Judge the worst answer: how to tell whether a clinical AI is safe to ship,” is published in The Renaissance Physician.
Dr. Hudes has a commercial interest in how clinical AI is built and evaluated.
Brian Hudes is a board-certified gastroenterologist and hepatologist with more than thirty years of clinical experience and a recipient of his specialty board’s thirty-year certification award. He built and ran a private gastroenterology practice for twenty years, then spent more than a decade in hospital-based gastroenterology, most recently as chief of gastroenterology and medical director of GI and endoscopy at a 550-bed Level I trauma center in Pensacola, Florida. He is an assistant professor of medicine at Florida State University College of Medicine.
Those two halves of a career shape the way he writes about medicine and clinical technology. Outpatient practice and hospital-based care fail their clinicians differently, and many systems are designed without a meaningful understanding of both environments.
Dr. Hudes has been building clinical software alongside his medical practice for decades. In 1995, during his gastroenterology fellowship, he co-developed one of the first Windows-based endoscopy reporting systems in the United States. He is now founder and chief executive officer of AiMOS Systems Corporation, where he is developing an AI-native clinical platform for outpatient and hospital-based care. AiMOS has been accepted into NVIDIA’s Inception program.
He writes about the architecture of clinical information, administrative cost growth, physician workforce shortages, board certification policy, and the widening gap between what clinicians need and what the health care technology industry builds. He also publishes The Renaissance Physician, an independent publication examining medicine, technology, physician workforce, and the structure of modern clinical practice.
