When Brian Hudes, MD, wrote “Why prompt engineering for physicians fails at 2 a.m.” on KevinMD in September, he opened with my research. I appreciated that. Then he disagreed with my conclusion. I appreciated that even more. Let me explain what I actually found, and why I think we’re both right, just about different parts of the same problem.
What my study showed
I tested six AI models (ChatGPT, Copilot, DeepSeek, Gemini, Perplexity, and OpenEvidence) on 77 pediatric board-style clinical questions. I gave each model the same questions twice: once as a single large prompt, once in smaller batches of 10. One model scored 38 percent in bulk mode and 83 percent in batch mode. A 45-point swing from prompt packaging alone.
My conclusion was practical: Physicians should learn to structure their inputs better. Hudes read the same data and reached a harder conclusion: A tool whose accuracy swings 50 points based on how you hold it has not been finished. He’s right. And so am I. Here’s why that’s not a contradiction.
Where Hudes is correct
His core argument, that we are engineering around a design defect and calling it clinical training, is among the most important things written about clinical AI on this platform this year. The analogy to anesthesia pin index systems is exact. Medicine spent decades moving safety burdens off individuals and into systems. Clinical AI is the first technology in a generation where we’ve voluntarily moved it back.
Real physician voices, twice a week
Free, and one click to unsubscribe.
He’s also right about 2 a.m. My prompt-structuring advice works at 8 a.m. with coffee. It degrades exactly when it matters most: on the twelfth consult of a call night, when a tired physician pastes in a long, complex case and trusts the fluent, confident answer that comes back.
What my study showed that his framing misses
Hudes frames the finding as binary: Either physicians adapt, or engineers fix it. I think the data support a third reading.
The same AI that scored 38 percent also scored 83 percent. The engineering is not uniformly broken. Something in the model’s architecture responds correctly to the right input structure, and that means the fix is not only possible, it has already been demonstrated inside the same model, on the same day.
What I’d add to his call to action
Hudes calls for engineers to take responsibility. I’d go one step further: Institutions deploying these tools should benchmark them under realistic clinical load, with standardized prompts, before go-live. My study is a template. The methodology is not complicated. It is just not being demanded.
Until it is, his advice and mine are both correct: Push vendors to build better, and teach physicians to prompt better. Not because the workaround should become permanent. Because right now, nothing else protects the patient at 2 a.m.
Vinson James is a pediatrician.
