Subscribe to The Podcast by KevinMD. Watch on YouTube. Catch up on old episodes!
Family physician, medical device inventor, and health care entrepreneur Tod Stillson discusses the article “AI governance in health care: Why physicians must lead the design.” Tod warns that the greatest threat to modern medicine is allowing artificial intelligence to be designed without physician oversight. He reflects on how hospital consolidation and private equity have previously redistributed power away from clinicians, and argues that AI could become the next structural layer that reduces physicians to mere units in a revenue architecture. Tod outlines five non-negotiable characteristics of trustworthy AI, emphasizing that technology must augment clinical judgment rather than impersonate it. He highlights the “accountability test,” reminding listeners that the responsibility for clinical decisions remains with the doctor, not the algorithm. Discover why physicians must move from being passive consumers to active architects to ensure that AI creates leverage with integrity.
Partner with me on the KevinMD platform. With over three million monthly readers and half a million social media followers, I give you direct access to the doctors and patients who matter most. Whether you need a sponsored article, email campaign, video interview, or a spot right here on the podcast, I offer the trusted space your brand deserves to be heard. Let’s work together to tell your story.
PARTNER WITH KEVINMD → https://kevinmd.com/influencer
SUBSCRIBE TO THE PODCAST → https://www.kevinmd.com/podcast
RECOMMENDED BY KEVINMD → https://www.kevinmd.com/recommended
Transcript
Kevin Pho: Hi, and welcome to the show. Subscribe at KevinMD.com/podcast. Today we welcome back Tod Stillson, family physician and health care entrepreneur. Today’s KevinMD article is “AI governance in health care: why physicians must lead the design.” Tod, welcome back to the show.
Tod Stillson: Kevin, it is always great to be with you and to be a part of your audience. I love the work that you do at KevinMD.
Kevin Pho: All right, so tell us why you decided to write this particular article and then tell us what it is about.
Tod Stillson: This article is really about the importance of governance when it comes to AI and health care. I think very few of our listeners will say they haven’t been touched by AI when it comes to how we deliver health care. It is dramatically transforming before our very eyes. But I think intuitively for us as doctors, we have worked really hard in our careers to build and develop trust. We build trust with patients, trust with our peers, and trust to use products, devices, and medical-grade products that lead to trust.
When it comes to AI, sometimes it feels like it gets thrust upon us. The real question behind it is who is behind it, and how trustworthy is it? I think that is important. I can speak of that just from an end-user standpoint, but I can also speak of it from the perspective of someone designing some of this software as well.
Kevin Pho: Tell us currently who you think is leading the charge when it comes to AI and health care. I think that it is such a transformative technology, and we are not really quite sure who is leading that charge. So in your perspective, what do you think?
Tod Stillson: I think there is no question that it is big corporations that are leading the charge. They have a lot of ownership over the raw product itself, from the GPTs of the world to the Claudes and others who have created these amazing LLM products that are transforming how our world operates. This is true not just in the world of medicine, but really in so many areas. I think again, these big corporations are leading them, whether they are East- or West-oriented, U.S. or China-based. That is irrelevant at this point to me, although that could be a whole topic of conversation today. At the end of the day, those big corporations have a vested interest in the use of their utility product. Think of it like electricity. The electric company loves whatever you are doing with the end use of their product, but they are providing the source of that.
So we have big corporations that are providing the source of the LLM for us. Then it is what the individual companies, sometimes larger, sometimes smaller like my health tech company, do with that raw piece of utility to transform it into a practical use that each of us can apply in our daily lives. So there is an opportunity here for us to harness it, for us to use it for good, and to have some influence over it as physicians. It doesn’t have to be passive. I read a note on LinkedIn recently that I really liked. This is an original thought, so I don’t remember who came up with it, but they said when it comes to AI and doctors, there are the resistors or skeptics, there are the blind adopters, and then there are the shapers. All of us are going to fit into one of those three buckets typically when it comes to how we feel about AI. I happen to be someone who is very interested in being a shaper. I think all of us, even if we are a resistor or skeptic or a blind adopter, should be interested in the governance of it, how it even got into our computer or into our life period, and who is going to be responsible for the output or the inputs that come with it.
Kevin Pho: Tell us what you mean specifically by AI governance and what are some ways physicians can lead that charge. I know we talked in previous episodes about how we lost that charge when it comes to electronic medical records, and we want to make sure it doesn’t happen again. So tell us what AI governance specifically is and what are some ways physicians can lead that.
Tod Stillson: When it comes to AI governance, I break it down into five characteristics. When you are evaluating an AI product, I think it is important for us to look at it in these five ways. First of all, you know that the AI product must augment our clinical judgment and support our decision-making, not impersonate it. So it is not just doing it for us. We can’t just passively let it do it for us. It must have explicit data boundaries. The other side providing us the raw information can’t have control over all the information, including who trains it and who is involved in how it is built. AI must be narrow enough to be reliable. It can’t do everything, Kevin. As much as we might like it to do a lot of things for us, it can’t be reliable if it is doing everything. It must be explainable and auditable. It also must leverage our clinical skills, not just be about doing something faster.
So when it comes to governance, it is all about what is behind or who is behind where that raw information comes from, number one. Number two is who or what is behind organizing the raw information into an envelope that you are actually using on your desktop or using in your clinical space. It behooves us not to just blindly accept that whoever is behind it has our best interest in mind. Here is an example I will give to you. You may be familiar with this. Nature Medicine magazine came out with an article recently from Mount Sinai and the University of Miami talking about the triage skills of raw LLM. Think about GPT. If a doctor got onto GPT today, described their clinical scenario, and looked for tips from GPT, doctors do this every day by the way, how accurate would that product be? Interestingly, the accuracy diagnostically of just raw GPT was somewhere around the 90th percentile. However, its triage ability was not so good. It overtriaged and undertriaged in a lot of things, and it ended up getting graded about a 50 percent or 54 percent triage rate.
So raw GPT or raw LLM is good, but for clinical application for us, you need somebody else who is building that raw information with some safeguards around it. Not to get too technical, but because I do medical engineering work as well, it is our RAG, our retrieval-augmented generation system, and our prompt engineering that are built around that raw information that comes from LLM. When you do that, Kevin, then you can take a succinct knowledge base, have that LLM use that knowledge base, have guardrails built into it, and dramatically improve the accuracy. As an example, my own triage system that we use just for acute infections with ChatRX is basically about 100 percent accurate in terms of our auditing on it. Is that because we are smarter than the folks at OpenAI? No. We have just taken it and organized it in a way and used that knowledge base with the RAG and prompt engineering to make it accurate. Then we test it, retest it, etc. Those kinds of things are what doctors need to know about what is going on behind the scenes to make something reliable and have the right governance.
I will tell you, I would rather have a doctor doing that than a health tech founder who went to school to be an engineer. I would rather have somebody who really understands a clinical space and can build that RAG or knowledge base. I would rather have somebody who can build that prompt engineering knowledge base that really accurately demonstrates that it is guideline-based and is using all the things that we are instinctively used to doing every day in clinical medicine, bringing it to bear upon that AI experience. So that is governance. At the end of the day when you apply that, you need to be able to take a look back and ask if it was accurate. Is it auditable? Can you identify where the clinical decision-making support happened? These are all really critical things regarding how doctors need to begin to look at AI.
Kevin Pho: Talk to us about some of the skills doctors need in order to take control of that governance. It sounds like you have a little bit of an engineering background, so you are able to blend what you know clinically with what goes on behind the scenes of these large language models. But I would say relatively few have that skill set. So how can doctors learn that? How can doctors blend the technological side with the clinical side in order to take charge of this AI shift?
Tod Stillson: First of all, one of the great things about doctors and our tribe, Kevin, is they are smart and they learn quickly. So it doesn’t take long for a doctor to get into the room or the cubicle where these decisions are going on, see the flow of information, and be able to very quickly provide the clinical knowledge that is needed for the engineer to act on without being an engineer. We begin to influence and identify exactly how we can use our practical knowledge in complement to that raw knowledge coming in through the LLM. Then we engineer it in the right package so that the sequence happens in the right way.
I will tell you, we are the experts on that sequence. We are the experts on how that knowledge and information flows. We are the experts on clinical reasoning, documentation protocols, and pattern recognition. That is what we do all day long, our nervous systems are trained to do it, and we are magnificent at it. So if we just can get ourselves in those positions of influence in those rooms, I think we can bring to bear the right influence in how those products are designed, implemented, and then applied in a real clinical space.
Kevin Pho: To use yourself as an example, give us a story about how you transitioned. I know you are a private practice family physician, and now you have created ChatRX, which is a health startup. Tell us about your own story in terms of how you learned those skills to do what you are doing now.
Tod Stillson: Well, first of all, it takes a lot of courage and effort. When I transitioned into retirement after 30 years of primary care, I identified a problem. That problem was access to health care in the rural space, especially for low-acuity problems like common infections. As we know all around the country, if you call for an appointment when you are acutely sick, it is 14 days or 21 days out, so it is hard to get in. You have to go to urgent care or the ER. Basically, I created a virtual urgent care that will allow patients to instantly be treated in about a five to ten-minute experience through a triage and treat system.
How did I conceive that? Because I had engineered it in my clinical office prior to retiring, where I managed 5,000 patients with my nursing staff. The phone triage system would know exactly the questions to ask and exactly where to put patients. My nurse doing the triage would know which patients needed to be seen for triage, which patients needed a message to me for phone medicine treatment, and which ones just needed some reassurance. That was very highly engineered in my clinical practice. So when I got out of that practice and had time and space to spend on this, I decided I was going to develop a prototype to facilitate that same experience but using AI as the lever. Then you develop an MVP product, a test market, and a commercialized product.
I have gone through that over a two-year process. How did I get there? I identified others in the engineering and development space who partnered with me to help me all the way along that process. Now I have a pretty good-sized team that helps me do that. We are scaling it out in a three-state area in the Midwest, including Illinois, Indiana, and Michigan. I learned a lot in that, Kevin. I was not afraid to jump into the fray, learn how to engineer, and learn how to engineer a clinical experience for patients, although I don’t write code. I let other people do that. At the end of the day, my target has always been those patients and how the patient is going to receive this.
So it takes courage, it takes money, and it takes an investment on my part to do that. Then it takes time and energy to develop a team around you to support it. Mine is a small example of what that can look like, but this is going on all over the country. What I am inviting doctors to do is come to the table. Don’t be that passive person who says somebody else is going to do it. Don’t say that is for those venture capital people who are really all about making money. I say let’s make a difference in this world that we live in. I call it a moral ambition. Let’s get involved for the moral ambition’s sake and make a difference for those patients and for our tribe now during this moment in time. It just takes a little bit of courage and effort to do that. Again, we are all smart enough to learn really quickly as you get into the middle of it.
Kevin Pho: How about within the constraints of a larger medical institution, because the majority of physicians are practicing in those employed settings. What are some ways that they can influence AI governance within the constraints of larger systems?
Tod Stillson: You are right, those decisions are often made at that level. By the way, those same large systems are struggling with that same governance question. So software or AI solutions are being brought into the medical environment. There are a lot of them, and that is the level that these large systems are struggling with regarding the governance piece. They ask how that interfaces with us and what responsibility our system has if AI support provides the wrong information to a doctor. For example, if there is a malpractice case that comes out of it because the wrong information was provided to a doctor. These are governance-related issues. By the way, the good news is there has not been an AI-related malpractice suit yet for a physician, but it will come eventually if they have been fed the wrong information.
How can you get involved in that governance-related system? It is via your committees in your institution. It is usually through the information technology committee, your EHR committee, or some other medical governance committee that you can step up to. You can let your voice be heard by the decision-makers who are involved in this and ask those critical questions about governance in the process. Make sure that those companies that they are joining arms with are doing due diligence to put the product in front of you. As the end-user doctor, it is OK to ask those questions. It is OK to ask credibility questions. We do that all day long. That is evidence-based medicine that we are used to.
If you were talking to a peer and they said they always do this in this clinical instance, you might be bold enough to say to them: “How did you come to that conclusion?” They might say it was because there was an article they read three weeks ago that said this is how they should do it. It provides the guidelines or the governance of where you come up with that information. That is how doctors learn to live and provide the best level of care. By the way, that piece of information that they learned three weeks ago may have been different from something they had learned five years previous when they were in training. So it is always evolutionary. For us as doctors, that is why we are continuous learners. That is why we are always pressing into these things to really learn and apply that knowledge for the sake of the patient and for the sake of our profession. The same applies with AI. This is not somebody else doing it. This is us having power or control over the governance of it.
Kevin Pho: We are talking to Tod Stillson, family physician and health care entrepreneur. Today’s KevinMD article is “AI governance in health care: why physicians must lead the design.” Tod, as always, let’s end with take-home messages. Do you want to leave with the KevinMD audience?
Tod Stillson: Kevin, I just wanted to point out that AI doesn’t carry malpractice insurance, but doctors do. The reason doctors do is that governance is critical. We are the decision-makers. We are the responsible party when it comes to the decisions that are made clinically for our patients. I sure hope we never get to the point where some large piece of software, AI, or design is autonomously making decisions for us when it comes to patient care. That is not how this should go down. How this should go down is we should be supported with our clinical decisions and given information that allows us to rapidly assimilate and make decisions. I am all for that.
But at the end of the day, we are the responsible decision-makers. Everything we can do to build, construct, and influence how that decision support lands in our lap to help us in those moments is critical to us when it comes to successfully integrating AI into real-life clinical medicine. So don’t be passive, and don’t get mad about it. Jump in and make a difference. In my case, I chose to build something that I think is pretty novel. But in a larger institution, like you said, there are plenty of ways for you to build the on-ramps and inroads to influence and have a voice in it.
Kevin Pho: Tod, as always, thank you so much for sharing your perspective and insight. Thanks again for coming back on the show.
Tod Stillson: Kevin, it is great to be with you, and I hope our audience can continue to provide the best level of care that they can. That is my passion. I know that is your passion for what you do, and using AI is going to be part of that.
