Vinson James has found something that deserves more attention than it will get. Testing six large language models on validated pediatric board questions, he reports that accuracy ranged from 38 to 90 percent when the questions were delivered all at once, and converged to 83 to 88 percent when the same questions were delivered in batches of ten. Identical questions. Identical models. The only thing that changed was how the work was handed over.
That finding is real, and most physicians using these tools have no idea it exists. His paper deserves to be read.
His conclusion is where I part company. He closes by telling us that when you ask an AI a clinical question, the most important variable in the answer may not be the AI at all, it may be you. He means it as empowerment. Learn the technique, get better results. I read the same data and reach the opposite conclusion.
A tool whose accuracy swings fifty points depending on how you hold it is not a tool that needs a better-trained user. It is a tool that has not been finished.
Real physician voices, twice a week
Free, and one click to unsubscribe.
We have run this experiment before
Medicine has spent three decades learning that safety systems which depend on human vigilance fail, and that they fail hardest on the people carrying the heaviest load. We stopped writing “use trailing zeros carefully” and removed them from the formulary. We stopped telling nurses to check concentrations attentively and built hard stops into the pump. We stopped relying on surgeons to remember laterality and made them mark the site, out loud, with the room stopped. Every one of those changes moved the burden off the tired human and into the system, because the tired human is exactly who the system exists to protect.
Now consider what James is asking. Break your input into smaller pieces. Query the same question more than once and compare. Treat any single answer as one data point. Every one of those is good practice. Every one of them also requires deliberate effort, on every use, indefinitely, from a physician who is behind.
Ask when this technique will be abandoned. Not at eight in the morning on a Sunday with coffee and a study set. At two in the morning on the twelfth consult of a call night, with the ED holding three more, when the model produces a fluent, complete, confident answer to the whole case pasted in at once. That is precisely when the unstructured input goes in, and precisely when the physician has the least capacity to notice that something in the middle of it is wrong.
A safeguard that degrades under load is not a safeguard. It is an explanation for what went wrong afterward.
The burden is on the design
If input structure determines reliability, then input structure is an engineering problem and it belongs upstream, in the product, decided once by people who understand the clinical work. Chunking, staging, redundant querying, disagreement flagging between passes: These are all implementable. None of them requires the clinician to know they are happening. The right number of items to hand a model at once is a design parameter, not a bedside judgment call.
The reason this has not happened is not technical difficulty. It is that the general-purpose chat interface was never built for clinical use, and we adopted it anyway because it was available and impressive. We are now generating a literature that studies how physicians should adapt to it, when the more useful literature would study how it should be rebuilt for us.
What I would take from his paper
Read James’s advice and follow it, because right now nothing else protects you. Chunk your inputs. Ask twice. Distrust a single confident answer. He is correct that this is teachable in an hour and correct that trainees are getting nothing.
But do not accept the framing that this is now part of the physician’s job. Every hour we spend teaching residents to compensate for a tool’s instability is an hour spent making a design defect permanent. Techniques get taught, absorbed into the culture, and eventually stop being questioned, and twenty years from now someone will be explaining to an intern that of course you break the case into pieces, that is just how it is done. Nobody will remember it was a workaround.
The most important variable is not the physician. It is whether anyone building these systems has treated reliability as their responsibility rather than ours.
Brian Hudes is a board-certified gastroenterologist and hepatologist with more than thirty years of clinical experience and a recipient of his specialty board’s thirty-year certification award. He built and ran a private gastroenterology practice for twenty years, then spent more than a decade in hospital-based gastroenterology, most recently as chief of gastroenterology and medical director of GI and endoscopy at a 550-bed Level I trauma center in Pensacola, Florida. He is an assistant professor of medicine at Florida State University College of Medicine.
Those two halves of a career shape the way he writes about medicine and clinical technology. Outpatient practice and hospital-based care fail their clinicians differently, and many systems are designed without a meaningful understanding of both environments.
Dr. Hudes has been building clinical software alongside his medical practice for decades. In 1995, during his gastroenterology fellowship, he co-developed one of the first Windows-based endoscopy reporting systems in the United States. He is now founder and chief executive officer of AiMOS Systems Corporation, where he is developing an AI-native clinical platform for outpatient and hospital-based care. AiMOS has been accepted into NVIDIA’s Inception program.
He writes about the architecture of clinical information, administrative cost growth, physician workforce shortages, board certification policy, and the widening gap between what clinicians need and what the health care technology industry builds. He also publishes The Renaissance Physician, an independent publication examining medicine, technology, physician workforce, and the structure of modern clinical practice.
