Your AI judges patients by how they write, not their symptoms [PODCAST]

Listen and followApple PodcastsSpotifyYouTubeMore apps · Browse all episodes

What if medical AI judges you by how you write instead of what’s actually wrong? A health care executive says it already does. Craig Hauben has spent 30 years in health care and the last several years researching AI, and he walks through studies showing that large language models trained on decades of human writing pick up on typos, misspellings, and emotional tone, then quietly route people away from the care they need. Close to 10 percent of the time patients who should have been seen were told not to come in, and women were hit hardest. This episode is based on his article “AI bias in health care reads the writer, not the symptom,” published on KevinMD. You will learn why the bias survives even after the obvious warning signs are scrubbed, why ChatGPT-style tools can carry the same blind spots into the exam room, and why trusting the AI too quickly is the real trap. Press play to hear the three questions every clinician should ask before trusting AI with a patient.

This episode is brought to you by ModMed.

Welcome to your new AI-Powered Practice from ModMed. We’re transforming specialty care by embedding AI Assistants across your entire workflow. Our all-in-one platform of EHR, patient engagement, practice management, and RCM helps reduce repetitive work while keeping you firmly in control.

Trained on de-identified data from nearly a billion patient encounters, this isn’t just smarter software. It’s a new way of working for specialty medicine, more efficient and more connected to the patient experience.

Start building your AI-Powered Practice at modmed.com.

VISIT SPONSOR → https://www.modmed.com/

SUBSCRIBE TO THE PODCAST → https://www.kevinmd.com/podcast

Transcript

Kevin Pho: Hi, and welcome to the show. Subscribe at KevinMD.com/podcast. Today, we welcome Craig Hauben. He’s a health care executive, and today’s KevinMD article is “AI bias in health care reads the writer, not the symptom.” Craig, welcome to the show.

Craig Hauben: Thanks, Kevin. Appreciate it. Happy to be here.

Kevin Pho: So let’s start by briefly sharing your story, and then we’ll talk about why you decided to share this particular article on KevinMD.

Craig Hauben: Sure, Kevin. Thanks. I’ve been in health care for more than 30 years, an executive all over, from insurance to working in hospitals and clinical settings. Today I run a company called Clutch, Clutch Health. We are a patient and member engagement platform that started in retail. So we have Amazon-type tech brought to health care to help drive behavior change in health care. And we’ve been doing AI for a decade, since the beginning. So we use AI to drive member and patient engagement, drug adherence, or clinical gaps.

And I’ve spent the last couple of years working deeply in AI and doing research in AI, and found out all kinds of interesting things that I just didn’t know, and I think a lot of people don’t know, and I think they’ll find interesting, and a little scary, and a little exciting, that I wanted to share and bring people along on their AI journey.

Kevin Pho: All right. So you tell us a story about one particular example in your KevinMD article, “AI bias in health care reads the writer, not the symptom.” Why did you decide to share this story, and then tell us about it for those that didn’t get a chance to read it?

Craig Hauben: Yeah. So I shared this story because I think there’s a lot of focus in general on AI and all the things that we can see. You may have seen in the last week there’s a lawsuit now against OpenAI for ChatGPT giving somebody some clinical advice that turned out not to be great. We tend to want to focus on those things a lot, but there are a lot of things that we don’t see. And that’s why I wrote this particular article.

The background of the article, and I think a thing to remember for all of us, is that all of the LLMs, all of the AIs, they are trained on decades and decades and decades of human writing. So there’s all kinds of things in human writing that we maybe don’t think about as much today that existed 40 years ago or 80 years ago. Those might be biases, prejudices, and other things. And those come through in how AI works with people in clinical settings.

The story I tell is really about how Nature, two years ago, ran a fairly simple study on some LLMs where they took traditional American language, and then they took African American language, and they ran tests. And without actually saying anything about somebody’s race or anything else, the AI automatically started to ascribe certain characteristics to the various test samples based on how they wrote.

MIT a year later brought that into the clinical setting, and what they did was they started adding in what we might call just everyday language or writing. Typos, misspellings, spaces, emotionality. And similar to the Nature study, in the MIT study, the AIs and the LLMs started to ascribe characteristics to individuals just on how they wrote their prompts. And what happened was close to 10 percent of the time, there were circumstances where whoever was writing the prompt probably should have come in for care and was told not to, based on these biases. And unfortunately, in this circumstance, it was women who got the worst of it.

But the real trap that comes from that, and this is really the end of the article, is that in the Nature study, they came back and said, “OK, so we will do a bunch of work to try to train the LLM from the outside to move away from these prejudices.” And what happened was the LLMs and the AIs did a great job of no longer using the words that they used to use, which clearly showed bias. But additional research showed that even underneath it, the bias persisted. And that’s something that we all need to think about. To me, that’s an important lightning rod for the bias that we can’t see. We see it every day in our lives, but AI has similar bias because of how it’s trained, and we need to think a lot about that, particularly as we think about using AI in clinical settings.

Kevin Pho: So, you mentioned a couple of studies. In an everyday setting, whether it’s a clinical setting or even a patient just querying a large language model about their symptoms, how would that look in a common scenario?

Craig Hauben: What that’s going to look like, if we just talk about the MIT study, for example, or maybe the eGFR study, which was similar bias but not AI bias, is that the patient isn’t going to get routed to the care that they need at the time that they need it. So that could be if it’s overt bias, for example, in a model where you can see it. That’s easy. There’s going to be a number that says, “Oh, somebody should have gotten a referral on day 12, but they didn’t get a referral until day 34.” You can test that, you can measure that.

What people need to think about in clinical settings is what are the biases they can’t see? Instead of looking at the success of the model and what the model does well, they need to look at what the model doesn’t do well. And if I were advising clinicians, particularly clinicians who are looking at bringing AI models into the clinical setting, they should be testing with sloppy language. They should be testing with real-life data, and they should be more focused on what it does poorly rather than what it does well.

Kevin Pho: So these studies were back in 2024 and 2025, and as you know, the iteration of these large language models is really exponential. So from what we know now, are these biases still present in today’s frontier models of AI?

Craig Hauben: I am sure that they are, because the data that is used to train the models isn’t evolving. You’re still essentially training models on the internet. You have to assume that in a way the AIs are like humans, that they have biases based on how they grew up, just like how humans have biases based on how they grew up. Until you can prove that they don’t, and I’m going to assume that they do, you just have to assume that they do, and you have to figure out where those biases exist.

And particularly as you think about how work is changing in clinical settings, where now AI is doing more work and clinicians are verifying the work, they need to pay particular attention to the fact that they’re verifying what’s not happening as much as what is happening.

Kevin Pho: So as a physician, and I certainly use AI in a clinical setting as well, what are some ways that we could overcome some of these biases? Is it simply a prompting issue? So what are some practical tips that you could share with clinicians?

Craig Hauben: I think the first thing that all clinicians, and all people who use AI, need to focus on is their trust of the system. There’s additional research that shows that people build trust in the AI over time, and that just happens by using it. Maybe when you first start using it as a clinician, you look at the first 10 or 20 or 30 things and think, “Oh, God, that looked good. That’s great.” And you start to trust it. What happens, regardless of the expertise and the intelligence of the individual person who’s using the AI, is that when they build trust, they stop looking at the AI, and they just let it happen.

So I think an easy thing to keep in mind is always to have a level of distrust. Because when you have too much trust, you will miss those things that maybe later on in a process become a little obvious, but because you’ve trusted it, you’re just not looking.

Kevin Pho: In your article, you give clinicians three questions to ask about any AI system. So what are those three questions?

Craig Hauben: The three questions for me are, first, what is it that it can’t do? Don’t just focus on what it can do. What can’t it do? Second is looking for harm. Instead of looking for harm that’s obvious, that it gave the wrong diagnosis, look for things that are missing: patients that maybe should be coming into the office, or diagnoses that you would expect to see more often. And then the third, I think, is the thing I just mentioned, which is to have a healthy level of distrust. Don’t run to trust, because as we run to trust with this technology, we don’t look deep and hard enough.

Kevin Pho: So in terms of what AI doesn’t do well, and we’re talking about biases, biases against women, biases against minorities in general, what are some areas that right now AI is still struggling with when it comes to a clinical setting?

Craig Hauben: I think a thing to think about in a clinical setting is we tend to evaluate the AI on a global average basis. How is it overall? On average, did it do a better job than humans at looking at a lab or at looking at a scan? The problem with that is that often the biases happen at the individual level, not at the average level. So I think a key thing is to continue to look a level deeper, not just, oh, did the AI overall do a better job, but where did the AI do a better job, and where did the AI do a poorer job?

Kevin Pho: Now, is there a variation between the frontier models, and of course we have physician-only models like OpenEvidence and Doximity. Is there variation when it comes to the biases and the answers in a clinical setting that physicians receive?

Craig Hauben: There is going to be a significant difference between any set of models. What you’re really hinting toward is that there are what I would call hyper-focused models, that they’re trained in very specific, narrow ways and very clinical ways. Those are actually the models that are going to get to superintelligence much faster than the generalized models. I think the key to remember is that even if a model is being trained in a narrow way, what is it being trained on? Is it being trained on, again, the same type of human documentation that has existed for decades, which holds those embedded biases, or is it a different type of data that maybe is less likely to have those kinds of biases?

Kevin Pho: Wasn’t there a recent Nature study that showed some of these narrowly focused trained models didn’t do as well when compared with some of the general models, the frontier models? Correct me if I’m wrong, but didn’t that study come out recently?

Craig Hauben: It did. It did. I don’t have the details in front of me to go through it in detail, but it did. Yes.

Kevin Pho: So what does that say about which approach we should take when it comes to the clinical setting in terms of general versus a more focused, really trained model?

Craig Hauben: You know, actually, I would go a little bit away from maybe what the data suggests that I should do. The way I still think about it is that the smartest thing to do is to imply and assume that there’s going to be bias in the model, whether it’s a generalized or a hyper-specific model. And the problem sometimes with either of them is that the data changes over time, and they move. I think you just have to assume that there is bias in it. You have to look for the things that you can’t obviously see, and in that, you’re more likely to get to a better outcome.

Kevin Pho: So what do we have to look forward to? As we mentioned before, the pace of evolution of AI models is just iterating so quickly. As physicians, tell us some of the things that we need to look forward to in the coming months.

Craig Hauben: Well, I mean, the whole world is changing. I took all of my research from the beginning of looking at AI deeply, and I wrote a book called The AI: Migration, which is really about how work changes over time, and I think over time is a very short period of time. Anyone can look at the news and see 20,000 jobs here or 100 jobs here or 500 jobs here.

I think in the clinical setting, the interesting thing is that where a physician was doing the work before, now the AI is going to do a bunch of the work, and the physician now needs to review that work. And that’s not that the work goes away, it’s a migration of who’s doing what, and I think that physicians particularly, or any clinician, need to be hypervigilant and hyperfocused on making sure that they’re giving everything they have to that overview work, because sometimes that work isn’t necessarily the sexy work that they went to school to go do. But it’s important work, and it’s where they get to use their knowledge and expertise in a way that maybe they didn’t in the past, when AI couldn’t have written the chart or done other things that were more menial. That’s going to free up physicians and clinicians to do more of that kind of high-value work.

Kevin Pho: We’re talking to Craig Hauben. He’s a health care executive. Today’s KevinMD article is “AI bias in health care reads the writer, not the symptom.” Craig, let’s end with your take-home messages to the KevinMD audience.

Craig Hauben: My take-home message is simply that you cannot verify what you can’t see, so you need to learn what they can’t do well, and you have to assume that the average does not necessarily represent the individual. I think if you do that, if you have that healthy level of skepticism, you’re going to be in a much better position to be able to find the right answer both for the patients and for the clinical settings.

Kevin Pho: Craig, thank you so much for sharing your perspective and insight. Thanks again for coming on the show.

Craig Hauben: Thank you. I appreciate it.

Listen and followApple PodcastsSpotifyYouTubeMore apps · Browse all episodes