Subscribe to The Podcast by KevinMD. Catch up on old episodes!

Join Mike Hines, an evangelist at Resolute Software, to explore the transformative impact of COVID-19 on the IT landscape. We discuss the rapid shifts in remote interactions, the emergence of technical debt, and the challenges faced by IT departments. Mike sheds light on innovative solutions, such as microservices and containers, and offers insights into building change-tolerant architectures for a resilient post-pandemic future.

Mike Hines is an evangelist, Resolute Software.

He discusses the KevinMD article, “How COVID-19 unleashed the power of microservices and container-based solutions.”

The Podcast by KevinMD is brought to you by the Nuance Dragon Ambient eXperience.

Ambient intelligence augments human capabilities to make our lives easier. The applications are many, especially in health care. Ambient clinical intelligence is offsetting the most pressing challenges in health care today, such as burnout, physician shortages, physician and patient dissatisfaction, and underperforming financial outcomes, by applying the technology to clinical documentation. 

The Nuance Dragon Ambient eXperience, or DAX for short, utilizes artificial intelligence and natural language processing to automatically document care. It securely listens to and captures the natural, clinician-patient encounter conversation unobtrusively, and turns that conversation into a clinical note for the clinician’s review and signature directly in the electronic health record. You just talk naturally, and DAX does the rest. 

DAX is being used by thousands of physicians across 30 different specialties nationwide. It has already won the Silver Stevie award in the health care technology category and was ranked #1 for improving clinician experience in KLAS’s top 20 emerging solutions.

VISIT SPONSOR → https://nuance.com/daxinaction

SUBSCRIBE TO THE PODCAST → https://kevinmd.com/podcast

RECOMMENDED BY KEVINMD → https://kevinmd.com/recommended

GET CME FOR THIS EPISODE → https://earnc.me/m2dyi7

Powered by CMEfy.

Transcript

Kevin Pho: Hi, and welcome to the show. Subscribe at KevinMD.com/podcast and get CME for this episode by clicking on the CME link in the show notes. Today we welcome Mike Hines. He’s an evangelist at Resolute Software. Today’s KevinMD article is titled “How COVID-19 unleashed the power of microservices and container-based solutions.” Mike, welcome to the show.

Mike Hines: Well, thank you very much, Dr. Pho. I appreciate it.

Kevin Pho: So tell us your story and journey to where you are today.

Mike Hines: Well, my background is all technology. I spent about 10 years at Microsoft doing education software and 10 years at Amazon, mostly on gaming software. I’ve done a handful of startups in education and cutting-edge game technology, collected a handful of patents and then retired to a tree farm in Georgia. After retiring to a tree farm in Georgia and finding trees not exactly thrilling conversationalists, Resolute Software found me and gave me the opportunity to do a deep dive into the technology supporting health care. This is great, because I now get a chance to invent new processes, new services and new software that will help fulfill the promise technology has always had for health care and health care outcomes. For a guy who almost went pre-med as an undergrad, I had to jump at that chance, and that’s kind of how I landed here.

Kevin Pho: Perfect. So you had a background outside of health care; you mentioned Amazon and Microsoft. Now that you’re in health care, what’s specific about the health care field, the health care IT field, that makes it different from some of the other things you worked on in the past?

Mike Hines: Oh, absolutely. In the past, I’ve worked with people who are very much technology-forward. Amazon and Microsoft are all a bunch of technology people working in technology, and the game developers I worked with are all technology people working in games. Health care has been really different. There are a whole bunch of doctors working with patients, and administrators working with doctors, and IT, the technology side, has almost taken a back seat. It’s fascinating to see the conjoining of clinical work and technology work, affecting patient outcomes, and I think we’re at a fascinating juncture where that’s possible.

Kevin Pho: So a lot of us physicians often don’t know what goes on behind the scenes on the technology side, and when I talk to a lot of tech specialists, they don’t know what goes on on the clinical side either. What’s something you want clinicians to know about the tech side of health care that they ordinarily wouldn’t know about?

Mike Hines: Actually, this ties in really well with one of the reasons I wrote the article. We have a whole bunch of unsung COVID heroes out there. First responders deserve every bit of credit they get, but health care IT departments made some miracles happen almost overnight. There was a price to pay for some of the miracles they pulled off, and my article calls out those miracles and the price they paid. But I’d really like clinicians to understand that your IT staff doesn’t hate you. They’re not doing any of this to get back at you. The IT staff have patients too, and sometimes their patients have good days, and sometimes their patients have bad days. Understanding will get you a lot farther when something doesn’t work on the IT side.

Kevin Pho: So let’s talk about your KevinMD article, “How COVID-19 unleashed the power of microservices and container-based solutions.” You talked about some miracles behind the scenes when it comes to IT during COVID. Tell us more about that, as well as your article.

Mike Hines: Absolutely. It all starts with the backstory to COVID. Up until 2021, a lot of IT staff had been staffed and budgeted to make sure that things like scheduling servers and billing servers worked well for the staff and the clinicians, and they were getting into electronic medical records, EMR management. Then COVID hit, and all of a sudden there’s a mandate to reduce physical contact and do social distancing in health care. Seriously, Kevin, imagine trying to get your car repaired when you’re not allowed to let the mechanic into the room or into the garage. That’s ridiculous. But IT, the doctors and the tech staff worked together to figure out how they could do a lot more of these things remotely.

Certainly there had been some prior effort in getting remote measurements out there, but that really ramped up, from older tech like cholesterol testing, colon cancer screening and sleep apnea testing to more real-time pacemaker monitoring, glucose monitoring, body temperature reporting and blood pressure collection. That all got really accelerated. Now, all the data collected by the remote monitoring, well, a patient can’t just bring that into the clinic. There goes social distancing, right? So it’s got to be entered online. Well, enter your local IT staff. If you have a chain of clinics, you’ve got to develop a web page, a way for patients to enter the data and a way for that data to get injected into the electronic medical record, and it’s all got to be done within HIPAA compliance. That’s a whole other story, IT and HIPAA compliance; we’ll write that article later.

But overnight, the IT department created these pages and discovered that, you know what, sometimes patients aren’t the best at actually recording all that data and reporting it online. So they had to go an extra step. They had to help patients connect those devices to their home networks, and then get those devices on the home networks to call back to the clinic, get into its network and get into its electronic medical record database to actually enter that data into the EMR. That’s huge, because up until 2021, the only things these server stacks had done were manage billing and appointments. All of a sudden, that same infrastructure was pressed into service managing thousands of connections from devices on remote networks, all wanting to insert data into your database. It’s no surprise that it was stressful for a lot of network infrastructures that existed to do billing and scheduling.

Oh, and by the way, this is the definition of technical debt: when you skip critical digital infrastructure improvements just to deploy critical technology. When I spoke to IT departments at the HIMSS conference in Chicago, a lot of the IT administrators I talked to said they wouldn’t have deployed it this way if there really hadn’t been that emergency. But deploy it they did. They got the miracles done, it worked and everybody loves the remote monitoring. Nobody wants to give it up, so they can’t go back, undo everything and then build out the infrastructure correctly. This is the new normal, and it’s not going away, and they’ve got to try to replace a flat tire while the car is still moving.

Kevin Pho: So take us into that time during COVID, when you were working, of course, in health care IT. Give us a little story or an anecdote that really illustrates how you had to deploy some of these technical solutions before you were fully comfortable doing so. Just give us a little anecdote from your world during that time.

Mike Hines: The big challenge is the migration of architecture technologies on the back end. When you only had one or two systems to maintain, prior to 2021, you could write them all on top of the same collection of server technologies. You have a server program that accesses scheduling, then a second bit of code you add on that does billing, and you keep adding parts to this one piece of server code, and you end up with what we call monolithic code: one gigantic chunk of code that’s trying to do half a dozen different things. As long as those things aren’t particularly stressful to the system, and as long as they’re all similar, you can run your website, scheduling and billing all off the same server and the same server stack, and it’s not really that stressful.

Asking remote devices on different networks to connect to your network and make direct database contributions of their data is a huge difference, and it’s a big weight. Normally, outside of COVID, you would want to break some of those services apart so they run independently of one another, and each gets the resources it needs to run independently. But that kind of time and that kind of budget weren’t available during COVID, so people started building these extended contact-free services on top of their existing stacks, with predictable results. We have an IT staff that has just created these miracles, but on a house of cards, so to speak. So they would spend most of their time trying to keep the system up and running, and all of a sudden new deployments and new implementations became really problematic, because they were spending a lot of time restarting servers. And trust me, as a clinician, nobody understands why the server is going down. They just know they can’t access the medical records they’re used to accessing. So all of a sudden IT gets a call from a very angry clinician: “Where’s my data? Why isn’t it working?” These poor guys.

Kevin Pho: And trust me, I’ve been that angry clinician on the other side of that call many times, so it’s pretty important to get that perspective. Just give us a contrast. Under normal circumstances, how long would it take to implement a solution in an optimal setting, as you describe, and how much was that time compressed during COVID? Just to paint a clearer picture.

Mike Hines: Oh, certainly. Before COVID, you could take the time to copy one of the services in your monolithic stack to a smaller deployment, let’s say one that only contains that service. You could call it a microservice; we do call it a microservice. And we give it a little bubble to run in, a little container, that has all the required bits it needs to operate: a microservice inside a container. We can test it and make sure it gives us the same results as the existing monolithic service, and if we get the same results, we can create lots of little copies of this container with this microservice, depending on the load it’s getting. More load, you just create more instances of the containerized microservice. Next thing you know, you can stop the calls to that part of your monolithic service and transfer them to the individual microservices. When you get more traffic, you instantiate more instances of the service, and if one little bubble, one little container, fails and crashes, nobody’s going to know, because you have all these other instances that can share the load until that little guy gets rebooted, and the service goes on as usual. None of the other services is affected.

So you do that for one, and then you do that for all the services in your monolithic stack, and next thing you know, all your services are collections of containerized microservices, and you have a very resilient system that can take a lot of change in demand and traffic and suffer a lot of impact and failures without anybody actually seeing that anything’s gone wrong. This process can take six to nine months to tease out, depending on how big your monolithic stack is. The COVID compression turned that into “What can you do in the next 30 to 60 days? Because we can’t have lines of people coming in, all waiting to sit down in one of our beds and get their tests done. We have to go zero-contact. What can you give me by the time our patients need their prescriptions renewed?” Yikes.

Kevin Pho: So here we are, three-plus years after COVID. What’s the current status of the majority of health IT solutions? Are we still building on top of legacy architecture? Are we unwinding and doing things the way they’re supposed to be done? Just in general, give us a status report on what health IT companies are doing.

Mike Hines: I wish there were a majority. It’s really hard even to say there’s a plurality; maybe that would be 30 percent. It’s all over the place right now, Kevin. The problem comes back down to the heroics we saw people doing. A lot of that is still built on the old infrastructure, and because staffing is still built around scheduling and billing, they don’t have the people they need to go ahead and tease those services out into microservices and containers. And with their staff busier than ever trying to keep everything standing, deployments of new technology are slowing down and becoming much more difficult. So if you’ve asked your IT department to deploy a new service, or make an update, say, to the website, you’re probably getting a lot more pushback now than you did before COVID, for precisely that reason.

Now, for some of the larger institutions, your Kaisers, your Mayos, hiring a couple dozen people to actually solve that problem is a lot more realistic, so they’re probably farther ahead of the curve, and were probably already on the curve to containerized microservices when this whole thing started. It’s the medium-sized collections of clinics, your regional multiclinic operations, that are really suffering the most. They have the greatest need for that kind of resilience, but they would have to increase their IT spend or bring someone else on board to get it done. So it’s tough finding the way forward without actually committing additional spend or resources.

Kevin Pho: So what lessons have health IT companies learned from the pandemic? You said they were able to compress what normally would be a six- to nine-month deployment into about 30 to 60 days. In the future, do you feel new health IT deployments can be accelerated because of the lessons learned during the pandemic?

Mike Hines: Certainly they could be accelerated, but I think we’ve also learned that it’s a really terrible idea unless you really do have an emergency that threatens interruption of clinic services, and we’re paying the price for that now. Technical debt is hard. It’s kind of like high blood pressure: You don’t know anything’s wrong until you get some sort of catastrophic systems failure. Unfortunately, a lot of people are leaving their IT high blood pressure untreated, and when another thing like COVID shows up, it’s not going to work like it did last time, because they’re still sitting under that technical debt. So I think, if we’ve learned an institutional lesson out of this, it’s that miracles come with a price, and it’s time for us to pay the piper and get to the level where we have containerized services.

Now, the nice thing about this is that it was probably going to happen with or without COVID. COVID just accelerated everything and gave us a critical failure point. The reason this was going to happen anyway is that when you have an architecture of containerized microservices, adding new services is a lot easier and a lot less risky for the system. You can take a new service, put it in its own container and try it, and if it crashes and burns and is a horrible disaster, none of the other services you currently rely on for your day-to-day clinical work get affected. It doesn’t take the whole system down; it just takes down its own little container, and you get to troubleshoot and fix it, and when you deploy it again, you can scale the number of containers to the right load. It becomes a lot easier to maintain and deploy new services. So that’s really the lesson I want IT administrators, and clinician administrators, to take out of this.

Kevin Pho: So just to reframe it: The short-term benefit we gained from that rapid deployment during COVID is leading to some longer-term problems going forward, and without a capital injection for fixes, like Kaiser and Mayo are making, you anticipate further problems down the road.

Mike Hines: Absolutely. It’s not an unknown phenomenon. Lots of companies will go ahead and deploy for the sake of competitive advantage before their infrastructure is really ready. They know they’re taking on technical debt when they do that. However, most of the companies that do that tend to be on the tech side of things, and they recognize the debt and have a plan to address it after their deployment is finished and their market advantage is established. I don’t see that happening in health care past the largest five hospital networks.

Kevin Pho: We’re talking to Mike Hines. He’s an evangelist at Resolute Software. His KevinMD article is titled “How COVID-19 unleashed the power of microservices and container-based solutions.” Mike, tell us some of the take-home messages that you want to leave with the KevinMD audience.

Mike Hines: Well, first of all, I’d like everyone who’s not on the IT staff of a health care organization to stop and recognize the role IT played in getting all of us through COVID and the social distancing mandates, with a really impressively minimal loss of humanity. So just take a moment and say thanks to your local IT staff. For administrators, the take-home message is: Recognize that technical debt exists, and then do something about it. If we don’t, we won’t have a miracle in the bag for the next COVID-like occurrence. For clinicians: Your IT team doesn’t hate you, and they haven’t gotten angry at you. Their patient is just having a really bad stretch right now, because it’s been overburdened with all of these other services that have to get done, and it falls over and needs to get picked up and restarted, and that’s going to cause a little instability on the practitioner side of things. But understand how that’s happening, and that this is one of the prices we’re paying. And clinicians, encourage your administrators to help IT help you.

Kevin Pho: Mike, thank you so much for sharing your perspective and insight, and thanks again for coming on the show.

Mike Hines: Well, Kevin, thank you very much. I appreciate it.