It was a quiet Sunday morning, coffee on the desk, and I was doing what most residents do at some point in their training: grinding through multiple-choice questions (MCQs) for my in-training exam. I had been using large language models (LLMs) as study partners for weeks. ChatGPT to walk through a tricky electrolyte case. Gemini for a second opinion on a renal physiology question I couldn’t square. DeepSeek when I wanted yet another voice in the room. Like most physicians my age, I had developed a casual confidence with these tools. I treated them roughly the way I treat consultants: different specialists, different styles, broadly reliable.
That Sunday, I pasted a batch of 100 pediatric MCQs into ChatGPT in a single block to power through them quickly. An hour later, slightly suspicious of how confident the model had sounded on questions I knew were hard, I pasted the same questions into the same model, but this time in batches of 10. The answers were different. Not subtly different. Strikingly different. The model that had answered confidently and incorrectly on a question about acute kidney injury management when given everything at once answered correctly when given the question in a smaller batch. It wasn’t a one-off. It happened across questions, across topics, across models.
I stared at my screen and realized that the tool I had been trusting to help me prepare for boards was answering differently depending on how I handed it the work. I had been asking the wrong question about AI in medicine for two years.
The question we keep asking, and the one we don’t: The medical literature on large language models is now enormous. PubMed indexes thousands of papers benchmarking ChatGPT, Gemini, Claude, and others against USMLE exams, board questions, and clinical vignettes. The framing is almost always the same: Which model is best? Which one passes the boards? Which one most closely matches expert clinicians? These are reasonable questions. They are also, in a meaningful way, the wrong ones. In a study my co-authors and I published in Scientific Reports earlier this year, we tested six publicly available LLMs (ChatGPT, DeepSeek AI, Gemini, Microsoft Copilot, Perplexity AI, and OpenEvidence) on 77 validated pediatric residency-level MCQs. We tested them under two conditions: all questions delivered simultaneously, versus delivered sequentially in batches of 10. The questions were identical. The models were identical. Only the delivery structure changed.
Real physician voices, twice a week
Free, and one click to unsubscribe.
When questions were presented simultaneously, model accuracy ranged from 38 to 90 percent. That is not a benchmarking result. That is chaos. A 52-percentage-point spread across leading models tells you that whatever these systems were doing, it was not reliable clinical reasoning. When the same questions were presented in batches of 10, accuracy converged to 83 to 88 percent. The inter-model variability collapsed. The statistical differences between models disappeared. The “best” model and the “worst” model became indistinguishable. The implication is unsettling and clarifying at the same time: When we publish a paper saying “Model X outperforms Model Y on medical reasoning,” we may actually be publishing a paper about how the authors structured their prompts. The model wars in medical AI may, in significant part, be a “prompt-structure” war that nobody noticed they were fighting.
What this means for the bedside
Most physicians I know are now using LLMs in some clinical-adjacent capacity. Drafting patient instructions. Summarizing literature. Generating differentials for a confusing case. Pre-rounding on a complicated patient. Residents preparing for in-training exams and boards. We do this without much thought to how we structure the input. We paste in the whole case, the whole list, the whole question set, and expect the system to handle it.
Our data suggests that habit is costing us reliability we don’t even know we’re losing. What is the remedy?
- Smaller chunks, not bigger ones: When I ask an LLM to help me think through clinical questions or work through practice MCQs, I now break the request into smaller pieces. Five differentials at a time, not twenty. Ten questions at a time, not a hundred. One complex case fully worked through, not a list of three. This feels slower. It produces more reliable output.
- Stop trusting the “best model” narrative: I no longer pick models based on which leaderboard they topped last month. The performance differences between frontier models, under realistic clinical input conditions, are smaller than the performance differences within a single model across different prompt structures. Spend your attention on the prompt, not the brand.
- Treat single-shot answers with appropriate skepticism: If you ask an LLM one question and get one answer, you have one data point. The same model, asked the same question structured differently, may give you a different answer. This is not a bug to be fixed; it is a property of how these systems work in 2026. Build redundancy into how you use them, especially when you are using them to learn.
What this means for medical education
This is where the finding becomes most urgent. We are training a generation of residents and medical students who will use these tools throughout their careers. Most of them (most of us) have received no formal instruction in how to interact with them. We assume the interface is intuitive. We assume the model is the variable. We assume that if we just pick the right tool, the rest takes care of itself.
None of that is true.
The story is more pointed for trainees. I have started incorporating a more useful frame: AI as a fallible collaborator whose reliability depends substantially on how you engage it. For program directors and educators reading this: This is teachable. The return on a one-hour investment is enormous.
What we got wrong, and what comes next
I want to be honest about the limits of what we found. Our study used MCQs, which are a specific and constrained format. Whether the same input-structure effects hold for open-ended clinical reasoning, for differential generation, for treatment planning: These are open questions. We are now extending the work to a broader panel of models and to non-MCQ tasks.
But the core finding feels durable enough to act on now. The variable we have been ignoring (input structure) is at least as important as the variables we have been obsessing over.
That Sunday morning, what started as a study session became a study. What I actually learned is something I now teach others: When you ask an AI a clinical question, the most important variable in the answer may not be the AI at all. It may be you.
Vinson James is a pediatric resident.
