Listen and followApple PodcastsSpotifyYouTubeMore apps · Browse all episodes
Doctors are adopting tools that answer clinical questions, and some give inconsistent answers or invented patient data. Neil Panchal, an emergency physician and cofounder and chief medical officer of Longevitix, a clinical AI decision support company, wrote a quick screen for colleagues asking which tools to trust. This episode is based on his article “4 questions to ask before you trust clinical AI tools,” published on KevinMD. You will hear whether a tool shows its sources, gives the same output for the same input, admits what it does not know, and reaches the right person without alarm fatigue. He explains how a lab panel from overseas shows the danger of a tool that quietly converts units. You will also hear why benchmarks can mislead, why he treats these tools like a first-year resident, and why they need monitoring after deployment. Press play to hear why he says physicians should help shape clinical AI instead of waiting for technology companies, and why an impressive demo is not a safe clinical system.
If this episode helped, follow The Podcast by KevinMD on Apple Podcasts or Spotify, and leave a rating.
Partner with me on the KevinMD platform. With more than 100 million impressions a year and nearly 750,000 followers, subscribers and email recipients across my channels, I give you direct access to the doctors and patients who matter most. Whether you need a sponsored article, email campaign, video interview, or a spot right here on the podcast, I offer the trusted space your brand deserves to be heard. Let’s work together to tell your story.
PARTNER WITH KEVINMD → https://kevinmd.com/influencer
SUBSCRIBE TO THE PODCAST → https://www.kevinmd.com/podcast
Transcript
Kevin Pho: Hi, and welcome to the show. Subscribe at KevinMD.com/podcast. Today we welcome Neil Panchal. He’s an emergency physician and physician executive. Today’s KevinMD article is “4 questions to ask before you trust clinical AI tools.” Neil, welcome to the show.
Neil Panchal: Thanks, Kevin. Glad to be here.
Kevin Pho: All right, let’s start by briefly sharing your story and then telling us why you decided to write this article on KevinMD.
Neil Panchal: Sure. Neil Panchal. My initial career was in emergency medicine, and then my clinical practice transitioned from the hospital setting to outpatient preventive health care using house calls, and I still practice that way now. Along the way, I was involved in several clinical informatics roles and trained with AI and health care through Stanford Medicine, as well as agentic AI through the Harvard Data Science Institute, and it really brought me to the intersection of frontline medicine and technology all around. Now I serve as the founding chief medical officer at Longevitix, which is a platform for multimodal AI clinical decision support for physicians.
I wrote this article really for my friends and colleagues, Kevin. They kept coming to me with the same problem. Thankfully, they’re starting to adopt and use clinical AI and decision support tools, but they kept seeing the same problem, which was inconsistent answers or hallucinated reports, and even in some cases patient data that didn’t actually exist. And so they came to me asking, “Neil, how do I tell a good product from another, and what should I be looking for?” So really, it helped me build these tools as initial screening items that colleagues can use. There are many, many more, but we can get into those four as we do.
Real physician voices, twice a week
Free, and one click to unsubscribe.
Kevin Pho: So in terms of how your colleagues are using AI, there are so many different ways, so many different tools. So when it comes to the clinical decision support space, are we talking about tools like OpenEvidence, UpToDate, Doximity? Are we talking about things like just generic large language models, Claude, GPT? Like, how are your colleagues using clinical AI tools today?
Neil Panchal: Yeah, great question. It’s certainly the former tier that you explained, right? The OpenEvidence, Doximity Ask, etc., because those are initially intended to be clinical decision support tools. I think the more frontier LLM models were not designed to be a decision support tool, but have the strength and possibility to do so. So a lot of our colleagues have kind of jerry-rigged their own way to do it, but at the end of the day, security-wise and whatnot, I would suggest sticking to the initial systems that have been built for clinical decision support.
Kevin Pho: All right. So your KevinMD article is “4 questions to ask before you trust clinical AI tools.” So tell us more about that article.
Neil Panchal: Yeah, I broke it down into four different steps to ask before you start using a type of clinical decision support tool. And these are, I think, generalized steps you should think about in anything in AI these days. It’s more relevant to physicians because of the way we think and how we practice. So one is traceability. Second is reproducibility. The third point we talked about is the defined limits, and the fourth is escalation logic.
And I’ll go through them one at a time just to understand what that means, and it’s not anything challenging. We do it all the time in medicine. Traceable outputs means that every output should link back to its original source, essentially showing you the work of how it came up with that output. In medicine, we ask our trainees, and we talk about presentations all the time, right? Like, “Tell me about your differential diagnosis and your thought process.” So we need to see that for reliability.
The second is reproducibility. And this means if you have an identical input, it should produce an identical output each and every time. And a tool that has an output that drifts each session is not what we need in medicine, right? Because that’s going to change continuity of care. Even if it looks like a beautiful answer, it’s not the right tool for us.
The third, I think, is the most important, which is what we call defined limits, or the awareness of the model. It should know what it doesn’t know, and it should tell you that it doesn’t know. This happens frequently because sometimes there are inputs that are missing, or they’re contradictory and conflicting, or they’re outside the base the system was initially built on. So instead of just filling gaps and giving a really plausible answer, it should be able to say, “I don’t know, and this is why I’m escalating it to you and back to you, doc. Tell me how to take the next steps.”
We had a recent example where we had a patient come from overseas, right? And they had a laboratory panel done there. I’m sure this happens all the time, and you see this in your practice as well, Kevin, where the units and the reference ranges are different on different continents. And if you put it into a single model, sometimes it’ll quietly just convert and use references that it’s comfortable with and it knows. But that’s dangerous, because it didn’t even flag to you that it did so, and you might just interpret it the way you do, and obviously it has downstream impact on the patient’s care. So that failure mode is what worries me the most, but we should keep that in mind.
And the last one is escalation logic. This brings me back to emergency medicine, or, Kevin, you’ll remember your training days on overnight call when a nurse calls you for every abnormal result, right? We don’t want that. We don’t want something that’s going to produce alert or alarm fatigue. We want the right information reaching the right person at the right time. So it’s not every single data point. It shouldn’t interrupt me unless I’m the right person to receive what matters most, right? So those were the four points we talked about.
Kevin Pho: So, in general, and we’re speaking in the middle of September 2026, what is the state of clinical AI tools today? Because I see ads for all of them. They each point to a specific benchmark saying they’re the best, right? So as a physician who doesn’t follow AI regularly, we’re a little bit confused in terms of which tool is the best, which tool satisfies those four questions, because you could just maximize benchmarks and just teach to the test, so to speak. So for a physician who’s not versed in AI, how can we tell which tool to use?
Neil Panchal: Yeah, it’s a great, great question, and we’re at an evolution transition point, which is a beautiful place to be, but it’s also a place to have caution, especially in medicine, right? Because it impacts our patients’ lives, and that’s what’s most important here. What that means is these models, or excuse me, not the models, the platforms, are very, very good. They enhance our ability to perform what we need to do very well, but they’re not perfect yet. And that’s how I would advise anyone, to come in with caution. That doesn’t mean don’t use them or ignore them, but still use them with caution.
I think of them as like a resident, right? There’s a different progression between a first-year and, let’s say, a final-year resident. You have more caution in your first year, and you’re going to double-check their work more often until they’ve kind of graduated to that final year of residency training, where they’re a lot more independent and you still just check on them once in a while. So I think that’s the way to go. I’m confident these platforms are going to get better and better as they learn.
The benchmark question is really unique and important to understand, and we’re very good at this as physicians and scientists in the health care space: You have to know what the endpoint is that’s being studied compared to what the platform was designed to do. And I think that’s where a lot of this confusion really happens. Recent papers have come out that say frontier LLMs performed better than some of these more clinical decision support tools. But if you dive deep, those benchmarks are kind of biased and skewed. And who better than us as physicians to know how to read a paper and understand confounding bias and understand what endpoints really are?
And so we should get familiar with how that looks from an AI standpoint and these AI tools’ standpoint, and understand that so far the benchmarks are very much against Q&A test questions, and we have some, and a decent amount of, synthetic or simulated cases. But the next best step is going to be a human patient base and real clinical data that’s going to be ingested and tested against, and I think that’s where we’re going to get some promising answers.
Kevin Pho: So tell us more about that study. I think you’re referring to that paper in Nature, right, that said the frontier models perform better than the clinical AI. So for someone, again, who’s not versed in AI, who just takes those results on face value, tell us in simple language, why were you suspicious of what those benchmarks were looking at? Why were you suspicious? It sounds like you’re suspicious about the findings of that paper.
Neil Panchal: Yeah, I think one easy example is that the length of the answer for some of the frontier models was longer than some of the clinical decision support models, right? And it’s a good example of how a judge, let’s say we’re using an LLM or AI model as a judge, interprets how comprehensive or accurate that answer might be because it’s more robust from the length perspective. But that doesn’t necessarily equate to accuracy or clinical applicability or what you can do at the bedside with your patient, right? So that’s just one example.
There were a couple of other benchmarks that I think are skewed, but some of the other benchmarks are not really applicable at the bedside. They might say, “This was a nicely written answer,” but OK, how’s that going to help my patient’s blood pressure at the end of the day, right? So that’s how we have to look at it, and that’s how we have to read these papers as well.
Kevin Pho: So for the physician who wants to choose an AI model, a clinical decision support model, just tell us the questions they need to ask before choosing the right one, because there are so many options. I mentioned a few earlier on this episode. Also, there’s a frontier model for people who go that path. So tell us the path they should go through, the decisions that they need to make in order to choose the right one for them in a clinical setting.
Neil Panchal: Yeah, it’s an excellent question. Go through the demos, use it, experiment with it, ask the challenging questions back to say, “How does your tool or platform perform when it fails?” I think that’s a very good way to assess it, and then see how it actually applies to a clinical problem you’re trying to solve, right? Everybody has a different use case, and if it doesn’t match your specific use case, it’s not the right tool for you. It could be a very good tool for somebody else. But that’s the best way, I think, to find a match first. And then overall, over time, we’ll start to see less nuanced or siloed tools and more generalized tools that’ll be more powerful and give you more of a single platform to give you multiple outputs that are going to be helpful.
Kevin Pho: Are we going to come to a point where these models are evolving so quickly where they’re essentially interchangeable? Like, it doesn’t matter which clinical AI model we use, because if you look at the frontier models, you have Claude Fable, you have GPT Astra. Eventually, they perform similarly on the top benchmarks. So from a clinical AI standpoint, are we going to eventually come to a point where it really doesn’t matter which tool we use?
Neil Panchal: Generally, yes, because they’ve gotten so good. The one thing I’ll say, which is different than what we do in medicine generally, is with these models, we have to have monitoring post-deployment, because the behavior can change, and I don’t want to say it evolves, but it will be different. And I say that because what we are used to in medicine is our input and output being the same each time, a clinical decision rule or a risk calculator, right? If you’re in pediatrics, you’re using PECARN, or in adult medicine, Wells criteria, PERC. But every time you put in the same inputs, you’re going to get the same outputs. It’s not the same way with, like you said, these powerful and evolving models. So that’s what we have to keep our skepticism about.
Kevin Pho: So tell us, what do we have to look forward to? I normally say in the next six to twelve months, but six to twelve months is a long time in this generative AI world. So how about in the immediate future? What do we have to look forward to in the clinical AI space?
Neil Panchal: Yeah, there’s a lot, and I’m optimistic about it, with caution, of course. Generally in health care, we’ve got profound use cases, right? I think drug discovery is a powerful place. We’ve seen that historically already. I think digital twins are really promising, to simulate two parallel, like I said, simulations for a single person, oncology, and a lot of these rare diseases, genetic scenarios. We’re going to see a lot of opportunity there.
But just generally, on an everyday basis, I think multimodal AI, which is what I’m working on specifically, is powerful because we’re inundated with data. I think we’ve gone into a great space where we’re producing a lot of diagnostics that are helpful, but docs like you and I on an everyday basis, I’m sure in your practice as well, it’s hard to harness all of that in one headspace for each and every one of your patients. So if we have a tool that does that really, really well, it can become a great assistant to augment your practice. I would start just there and look forward to that being a solid tool in the next, call it, two to three months, and then more and more coming up in the six-plus months.
Kevin Pho: How far away are we from multimodal AI? So, as a primary care physician, if I just throw an EKG, various imaging studies, an echocardiogram into a clinical AI model, it should be able to interpret it without any text.
Neil Panchal: Yeah, absolutely. I think we’re already there. We’re already there to interpret it at minimum, but even better, we’re able to make connections, and that clinical reasoning part is what I’m super excited about in, call it, now slash the next couple of months, is to make the connections that it’s supposed to make, to say not only is this cardiovascular systems biology, EKGs, echo, whatnot, but how do we connect it to that woman who had preeclampsia so many years ago, and her risk of cardiovascular disease is different, right?
So if that information exists, AI can do that. You and I, if the information is in front of us, either we’re looking for it, we’re asking somebody to find it for us. If it’s not in front of us, we can make those connections, right? So I think that’s very, very exciting, the clinical reasoning that goes into multimodal AI.
Kevin Pho: We’re talking to Neil Panchal. He’s an emergency physician and physician executive. Today’s KevinMD article is “4 questions to ask before you trust clinical AI tools.” Neil, we’ll always end with your take-home messages to the KevinMD audience.
Neil Panchal: Thank you. I’ll narrow it down to three take-home messages. I think, don’t confuse an impressive AI demonstration with a safe clinical system. In medicine, we need to be thorough. We need to be skeptical. We need evidence. We need transparency. We need to be able to see what it does when it’s wrong.
Second, I think we as physicians need to participate in this era. Don’t wait for the technology companies to come up with a solution for us. Historically, we’ve seen that things don’t go the way we want. And if we want the clinical outcomes to be the way we want to see them, then let’s partake in that. Let’s sit at the table in the rooms that we need to.
And lastly, I would say just don’t be afraid of using AI. Obviously, don’t blindly trust it either, but this is really going to enhance our ability to take care of our patients. That’s why we have these conversations. And patients are not caring about which tool you’re using. They’re caring that you’re going to be able to spend more time with them, understand their situation, explain it better, and treat them with the human care that we want to care. So use technology to leverage that.
Kevin Pho: Neil, thank you so much for sharing your perspective and insight. Thanks for coming on the show.
Neil Panchal: My pleasure.
Listen and followApple PodcastsSpotifyYouTubeMore apps · Browse all episodes
