The most consistent advice physicians receive about clinical AI is that we should be more careful with it. Verify the output. Treat it as a draft. Structure the input. Build verification into practice as the defining skill of the coming decade.
All of this is good counsel. None of it addresses the problem the same authors have already correctly identified.
Harvey Castro made the diagnosis on this site in May 2025, writing about the model context protocol: “The real limitation of clinical AI isn’t the model; it’s the context.” His examples were the blood pressure entered three minutes ago and the allergy buried in yesterday’s note, neither of which the system can see. That is not a complaint about model quality. It is a claim about what reaches the model, and in what form.
Hold that claim next to the guidance the field now offers clinicians, and something does not fit. If the limitation is context, no amount of physician diligence can repair it. Verification cannot supply data the system never received. Careful prompting cannot restructure a record the clinician did not build and cannot inspect. We have identified a problem in the plumbing and issued instructions to the people standing at the tap.
Real physician voices, twice a week
Free, and one click to unsubscribe.
The evidence keeps arriving
Vinson James recently reported that six large language models answering identical validated board questions produced accuracy ranging from 38 to 90 percent when the questions were delivered in bulk, and 83 to 88 percent when the same questions were delivered in batches of ten. Nothing changed but the packaging.
That is Castro’s thesis in experimental form. It says the arrangement of the input, a variable invisible to the person typing, moved performance further than the choice between leading models. James’s own conclusion is that physicians should therefore prompt more carefully. I would put it differently. He has demonstrated that a decision no clinician has been told they are making, one they receive no training in and no feedback about, determines whether the answer is right.
Why this failure mode is worse than the ones we know
Medicine has learned to engineer around human error, and the lesson was always the same: Remove the opportunity rather than exhort the operator. Anesthesia did not train providers to attach the correct gas cylinder more attentively; it built a pin index system that makes the wrong connection physically impossible. Neuraxial and enteral tubing were not addressed with a memo; they were given incompatible connectors. These are forcing functions, and they work precisely because they do not depend on anyone being rested or unhurried.
Clinical AI presents a harder version of the problem, and it deserves saying plainly. A wrong-side procedure announces itself. A misconnected line eventually declares what happened. A degraded model output does not. It arrives in the same confident register as a good one, in the same format, at the same speed. There is no roughness at the edges to catch, no moment where the physician senses that something has gone wrong.
Verification is offered as the remedy. But verification against what? A clinician checking an AI-generated differential against their own knowledge will catch the errors they were already equipped to catch, which are not the ones that matter. The errors that matter are the omissions, the finding that was in the record and never made it into the context window, the item that fell out of the middle of a long input. Those leave no trace in the output. The physician is not merely unaided in detecting them; they are structurally incapable of it.
We are asking clinicians to audit a process whose failures are invisible to them by design.
Where the obligation actually sits
Someone knows how these systems behave. The organization deploying a documentation tool knows how much text it can hold before retrieval degrades. The vendor knows what happens to a long input in the middle. Those facts are knowable, testable, and currently not disclosed to the clinician who signs the note and carries the liability.
That asymmetry is the actual scandal, and it is where the profession should be pressing. Not on training physicians to compensate, but on requiring that the compensation be built in and demonstrated. Chunking, staging, redundant passes with disagreement flagged, an explicit statement of what the system could and could not see when it produced a given output. All of it is implementable. None of it requires the clinician to know it is happening.
Castro is right that boards approve these tools before deciding what problem they are meant to solve, and right that no initiative should be approved without input from the clinicians who will use it. I would add one requirement to the list. Before go-live, the organization should be able to state what the system’s reliability does under realistic clinical load, and what it has built to hold that reliability steady. If the honest answer is that reliability depends on how carefully each physician structures the input, then what has been purchased is not a clinical tool. It is a liability transfer with a good interface.
The correct target
I am not arguing against verification, and physicians should keep doing all of it until something better exists. I am arguing against the drift in where we have placed the burden.
Every serious safety advance in medicine moved responsibility off the individual and into the system. Clinical AI is the first technology in a generation where the profession has moved it back, and we have done it voluntarily, out of enthusiasm, before anyone insisted.
The limitation is the context. Castro was right about that in 2025 and it is more true now. Context is not something a physician can fix from the keyboard. It is something that is either engineered correctly before the tool reaches the bedside, or is not engineered at all.
This essay is cited in the KevinMD record on artificial intelligence.
Brian Hudes is a board-certified gastroenterologist and hepatologist with more than thirty years of clinical experience and a recipient of his specialty board’s thirty-year certification award. He built and ran a private gastroenterology practice for twenty years, then spent more than a decade in hospital-based gastroenterology, most recently as chief of gastroenterology and medical director of GI and endoscopy at a 550-bed Level I trauma center in Pensacola, Florida. He is an assistant professor of medicine at Florida State University College of Medicine.
Those two halves of a career shape the way he writes about medicine and clinical technology. Outpatient practice and hospital-based care fail their clinicians differently, and many systems are designed without a meaningful understanding of both environments.
Dr. Hudes has been building clinical software alongside his medical practice for decades. In 1995, during his gastroenterology fellowship, he co-developed one of the first Windows-based endoscopy reporting systems in the United States. He is now founder and chief executive officer of AiMOS Systems Corporation, where he is developing an AI-native clinical platform for outpatient and hospital-based care. AiMOS has been accepted into NVIDIA’s Inception program.
He writes about the architecture of clinical information, administrative cost growth, physician workforce shortages, board certification policy, and the widening gap between what clinicians need and what the health care technology industry builds. He also publishes The Renaissance Physician, an independent publication examining medicine, technology, physician workforce, and the structure of modern clinical practice.
