Subscribe to The Podcast by KevinMD. Watch on YouTube. Catch up on old episodes!

Most physicians using AI on patient data have no idea what the real security risks are. Francisco M. Torres, an interventional physiatrist, and Purab Patel, a medical student with a programming background, argue that cybersecurity has become a patient safety issue in medicine, and that the AI pipelines physicians now rely on are more complex than most clinicians realize. This episode is based on their article “Navigating the cybersecurity challenges of artificial intelligence in medicine,” published on KevinMD. You will hear how changing a few pixels in a medical image can flip an AI diagnosis, why AI note-taking can miss clinically significant findings, and what questions to ask any vendor before trusting a black box with patient data. You will also learn why Francisco tells his kids that AI certification may matter more than an MBA. Listen for a grounded take on treating AI as a tool, not a truth machine.

Partner with me on the KevinMD platform. With over three million monthly readers and half a million social media followers, I give you direct access to the doctors and patients who matter most. Whether you need a sponsored article, email campaign, video interview, or a spot right here on the podcast, I offer the trusted space your brand deserves to be heard. Let’s work together to tell your story.

PARTNER WITH KEVINMD → https://kevinmd.com/influencer

SUBSCRIBE TO THE PODCAST → https://www.kevinmd.com/podcast

RECOMMENDED BY KEVINMD → https://www.kevinmd.com/recommended

Transcript

Kevin Pho: Hi, and welcome to the show. Subscribe at KevinMD.com/podcast. Today we welcome back Francisco Torres, interventional physiatrist, and a fourth-year medical student, Purab Patel. Their KevinMD article is “Navigating the cybersecurity challenges of artificial intelligence in medicine.” Francisco and Purab, welcome to the show.

Francisco M. Torres: Thank you so much, Kevin, for having us.

Purab Patel: Thanks for having us.

Kevin Pho: Perfect. So Francisco, thank you again for joining me. It’s great to see you again. Tell us how this article between you and Purab came together.

Francisco M. Torres: Well, this is fascinating. I have always been interested in new technology, and AI for me for the last year, a year and a half, blow my mind away. And I got lucky that his background is a programmer, and a very good one. And so we start talking about the benefits of AI.

As a matter of fact, we wrote another essay about it, because I always thought that I needed to narrow the size of the neural foramina when I was doing epidurals and things like that. But that information was not being told to us from the radiologist, because it was easy for them to say moderate, severe. So I thought, look, if we do the AI, they can measure that, and the radiologist doesn’t have to waste time doing that.

So that conversation started, but then he told me, “Hey, I’m very concerned about cybersecurity. Now we’re putting the patient data out there, not just because of their name or Social Security number, but also because of medical issues that can cause death if it’s done not correctly.” So that’s the way we came out with the article.

Kevin Pho: All right. And Purab, you’re a medical student, of course. You have a background as a programmer, certainly interested in hearing your perspective. For those who get a chance to read your article, just tell us briefly what it’s about.

Purab Patel: Yeah, so the main gap that we were identifying is the lack of awareness around security issues of AI amongst health care professionals, and the fact that these AI pipelines are so complex. There’s so many steps to the training of the models, the implementation, the actual use, and each of those points is a point of vulnerability where cybersecurity attacks can occur and where malicious players can take advantage. So we wanted to identify some of those issues and really open the eyes of providers to those issues.

Kevin Pho: And Purab, when you say AI, are we just talking about publicly available models like ChatGPT, Gemini, and Claude? Are we talking about models specifically geared towards physicians like Doximity and OpenEvidence? So when we say just AI models, just give us a sense of what we’re talking about here.

Purab Patel: So we’re talking about both. The more publicly available models are still in use by many physicians in independent practice. And of course, hospital systems have their own proprietary models, and both have similar but also different vulnerabilities. So we are considering both in this essay.

Kevin Pho: So tell us more, Purab, about some of the vulnerabilities to a non-programmer. A lot of this audience are non-programmers, so what do we need to be concerned about?

Purab Patel: Yeah, so several vulnerabilities. One is in the very beginning when you’re training the models with patient data, making sure that the confidentiality of the patient is respected, and that’s one of the key principles that it’s medical training. But when we train these models, we are submitting information to them, and there are ways for malicious actors to extract that information from the models through various prompts. So making sure that we de-identify the patient sufficiently that the information can’t be recovered.

Other issues are at the time of actual use of the model. So, for example, with image analysis and radiology, there’s a big issue around adversarial attacks where somebody can change a few pixels inside an image and the model will come up with the wrong diagnosis. And that obviously has downstream implications for management. So that was another issue that we considered.

And then one more was the fact that these models are so fragmented sometimes in the way that the pipeline is implemented, that a model may be trained in one country, may be implemented in another country, and so the data is spread across multiple different regulatory landscapes. And the laws may be different in each of those locations. So that’s something else that we consider.

Kevin Pho: And to be clear, the publicly available models like the GPT, Gemini, and Claude, they are certainly not HIPAA compliant. Is that right, Purab?

Purab Patel: Yes, they’re not.

Kevin Pho: So, Francisco, when you heard this, tell us what your thoughts were when Purab told you about some of these cybersecurity concerns for these AI models.

Francisco M. Torres: So I was in shock because, you know, you mentioned HIPAA compliance, but it’s not the same with AI. It goes further, because right now we’re putting patients’ lives at risk because the hacker can modify a pixel, like he said, and change completely the algorithm. About what he was just telling me, that there is a hospital where the intensive care unit is run by the AI.

Purab Patel: Yes. There was a recent news article about a hospital in the northeast of the U.S. where somebody died. And part of that investigation was the fact that the ICU was using AI tools to automate certain monitoring of ventilators and things like that. And so that was just one news article that came up lately.

Kevin Pho: Now, in terms of potential safeguards before physicians use any AI models, Purab, what do you recommend? What kind of questions do we need to be asking ourselves whenever physicians use these AI models? Because I know that we have some basic knowledge that, you know, we’re taught just never to give patient information to publicly available models because whatever we type in is being used to train on, but that’s very cursory, rudimentary. So tell us the type of things that we need to think about before we use these AI models for the first time.

Purab Patel: So I think first off is just understanding what the purpose behind the model was and why it was created. Was it created specifically for a health care purpose application, or was it created for a generic application and then adapted to health care? So that definitely plays a difference in the security considerations that went into the design and the model and whether it was built for the health care environment.

Secondly, I think it’s important to consider where the data is stored and where the data is processed. Sometimes there’s this concept of a black box where you just put in the patient information and get an answer back. What are the steps? What’s the chain that the information is actually traveling down? What’s the infrastructure that this company has in place? Maybe the company that provides a model may be reliable, but they may have other vendors in the mix that are not as reliable or have had more significant issues. So understanding the complete chain is, I think, important.

And lastly, the point of use, making sure that you always double check everything, not completely and blindly trusting the AI. Making sure that that trust is earned over time instead of automatically assumed.

Francisco M. Torres: And another thing I want to interject is, like, in the past, Kevin, doctors that went through their life as a doctor, they wanted to get an MBA to be directors or whatever. I think now that’s changing. I think it’s more important. I even tell my kids that you need to get certified and trained in AI because it applies to everything we do in life right now. So I think probably there’s going to be courses geared towards doctors, clinicians, to know how to identify those areas that are vulnerable.

Kevin Pho: Now, Francisco, you use AI just within the last year or so and fantastic in terms of how powerful it is. But after hearing some of these concerns that Purab has surfaced and you guys talked about, has that changed your outlook? Has that changed how you use these AI models?

Francisco M. Torres: You know what, what it has done for me is to be more educated and be more careful, but I do love the technology. I think, again, going back to what we talk about, the radiology, there was also a big study done where the AI read X-rays already read by radiology, identified like 40 percent of TL fractures. So I think it brings a lot of good, but we need to be careful. We need to rely on very good tools, tools that I hope the government too gets involved heavily.

You know, and I don’t know with you, Kevin, but when we buy an EMR system, I don’t go into the detail. How do they get trained? I just buy it, I don’t really get into that. But I don’t think we can do the same thing with AI right now. We cannot assume that everything is, the vendors are doing their job. I don’t think we can assume that anymore.

Kevin Pho: Now, Francisco, you have an interventional physiatry practice. Just give us some examples of how AI has changed your practice in the last few months.

Francisco M. Torres: Well, we don’t actually have a system right now, but I just see it around the hospitals right now. Like, for example, the return when we get the MRI report, you can tell there had been some AI type of intelligence behind it and the radiologist just confirming right now. And I think eventually that two specialties that are going to suffer probably going to be radiology and pathology, because it’s just pattern recognition. So as long as you have somebody supervising them, I just think that it just expedite reports, expedite care in the hospital. I have seen it, that even medications are delivered through AI and things like that. So I think that it’s positive in that way.

Kevin Pho: So I’m reading a lot in the news that a lot of these AI companies, it’s a race, almost like an arms race in terms of who can produce the biggest, fastest, most powerful model.

Francisco M. Torres: Yeah.

Kevin Pho: And they want to be first to ship, right?

Francisco M. Torres: Yeah, before all the consequences.

Kevin Pho: But in health care, Purab, there’s a lot of consequences if we ship too quickly in health care. Now, in that context, how can physicians be careful? Because there are going to be a lot of vendors who are pitching these AI solutions to them, and hospitals, they’re navigating how they can best use AI, and then they’re going to respond to demands from their physicians to better use AI. How can we proceed more cautiously in this new world when everything is literally changing on a weekly basis?

Purab Patel: Yeah, it’s difficult. And I think part of that difficulty is the fact that medical training takes so long, and so there’s a lag period. Somebody that enters, for example, radiology residency today is not going to graduate until five years from now, and the world is going to be different, especially with the innovations in AI.

So I think one of the key steps is doing your research before you start using a tool. Making sure that you know who developed this model, what the overall reputation of the company is, has it been tested in other domains before it’s brought into the hospital. And also there’s obviously going to be a certain period of time where there’s coexistence of humans doing a certain task and AI doing a certain task, where you can monitor the progress, the accuracy, until you finally get confident enough that you allow the AI to be more autonomous. So I think it has to be a gradual transition in that sense.

Kevin Pho: Now, a lot of physicians still aren’t using AI at all. They’re not in our world. And I’m sure you, as a former programmer, and a lot of your other medical student colleagues, they’re using AI on a daily basis. Just from your perspective with a background in programming, just give us some clear do’s and don’ts when it comes to physicians using these AI models for the first time.

Purab Patel: So I think one of the first don’ts is blindly trusting outputs. It’s been well established that AI can miss things, and the rate at which it misses certain findings or certain pieces of text is not proportional to the clinical significance of that. So there’s been reports that in automatic note-taking software, for example, sometimes very important points might be missed. So I think making sure that we are aware of that is definitely important.

And secondly, I think, from a programming standpoint, the pipelines, how these models are trained, are very complicated. But the basics to understand is that it’s essentially a large collection of data, and it’s an average of that data that the AI is using. So rare diseases, rare cases that are not represented in that training data, or patient populations that are not represented in data, those are areas where you don’t want to trust the outputs as much. So I think that’s an important consideration.

Francisco M. Torres: One thing, Kevin, that I propose, you know, in Florida, every time we have to renew our medical license, we have to take courses of medical errors, of domestic violence. I think we just need to advocate to have that as part of our requirement to renew our license. Some basic courses that at least we can identify potential problems.

Kevin Pho: So, Francisco, other than talking to Purab about the potential of AI, did you get any exposure through CMEs and through your colleagues in terms of how AI is being intersected with what other physicians are typically doing? How are you learning about the medical applications of AI?

Francisco M. Torres: Well, I have to say I have taken some of those courses that they advertise, but not related to medicine. This is more than regular AI, but I can probably extrapolate from there. Like, for example, I go to my setting, that’s what I learned from with my nephew, is that you want to protect your privacy. So if you tell your AI not to train with your data, that’s something that you can do it on purpose, but it’s not going to be a default.

And I think that is, you know, little by little getting educated as I use it just for my personal business. I think eventually it will be better to know, like, that, for example, again, going to radiology and having a report generated by the AI, I want to know if a radiologist supervised that AI or it was without supervision, like that case that you mentioned, that there were no supervision. So I think those little thing, eventually we’re going to have to figure this out as we go, learn as we go.

Kevin Pho: We’re talking to Francisco Torres, an interventional physiatrist, and Purab Patel, a medical student. Today’s KevinMD article is “Navigating the cybersecurity challenges of artificial intelligence in medicine.” Now, I’m just going to ask each of you just to share some take-home messages with the KevinMD audience. Francisco, why don’t we start with you, and then we’ll end with Purab.

Francisco M. Torres: I’ll just start saying that AI is a tool, and it is not a truth machine. Cybersecurity is now a patient safety issue and we need to be concerned. And we need to start asking our vendors tough questions, not be complacent, and also be skeptical of any black box that said, “Oh, that’s the way it works.” And also I think, I don’t like regulations so much, but I think that in this particular case, we’re going to depend on some government regulation like they do with medical devices and medications, which it’s going to have to happen.

Kevin Pho: I agree. Purab, we’ll end with you. Yeah, your take-home messages.

Purab Patel: I agree with a lot of those points. I think AI is definitely a powerful tool and one that will transform the practice of medicine, but one that we also have to be skeptical about, critical about, make sure that we don’t take at face value without actually analyzing its benefits and its risks.

Kevin Pho: Francisco and Purab, thank you so much for sharing your perspective and insight. Thanks again to both of you for coming on the show.

Francisco M. Torres: Thank you, Kevin.

Purab Patel: Thank you.