During the following year, in one of the hospitals, an algorithm will be automatically taken out of service. Either the sepsis model will drift below the performance level at which it was approved, or the monitoring process will detect the tool failing for one subgroup of patients even though the overall figures remain satisfactory, and the system, which was designed to do just that, will suspend the model and wait for a human to take action.
That moment was not a failure. The models do drift and the case mix changes, which shows that the governance program is functioning. What is interesting is what the record reveals afterwards. How long did it take before a specific human employee looked at the suspended model and signed a decision regarding it: to reactivate it, to keep it suspended, or to retire it? Was the time involved four hours? Or six weeks? Is anyone actually aware?
I have come to the conclusion that this interval is the single most honest figure in the field of AI governance, since very few people measure it. Refer to it as governance decision latency, which is the time elapsed between an automatic suspension and the following accountable, signed decision concerning the same system.
The reason is that it matters more than the figures which we do report. Almost all the content in a governance binder can be put together in the weeks leading up to a survey. Inventories can be compiled, policies drafted, charters ratified, and training modules assigned en masse on a Friday. I made that point in my previous column: Evidence of this kind can be written up, whereas evidence showing that governance in fact operated can only build up over time with dates attached. Decision latency is the clearest example of this kind of accumulating evidence. In order to have a figure for it at all, three things must have actually taken place: A monitoring threshold must have been triggered, the event must have been recorded at the time it was triggered, and a human must have signed a decision with a timestamp. None of that can be prepared retrospectively. A program that can give its median decision latency is one that has been running between surveys.
Real physician voices, twice a week
Free, and one click to unsubscribe.
People have been here before. It wasn’t by drawing up better chest pain policies that we improved cardiac care, but rather when we began to time the loop from arrival to intervention and then published the interval. Door-to-balloon time had an effect on behavior because it measured actual response rather than just intention. All clinicians are aware of the difference between a hospital that carries out mock codes and one that assumes the crash cart functions. There is now a need for AI governance to establish its own response-time discipline, and the interval from suspension to decision is where this should begin.
To measure it honestly you have to follow three rules. The clock begins when the suspension is fired, not when someone becomes aware of it, since discovery lag is a part of the problem, not an excuse for it. The clock ends when there is a signed and identifiable decision, not at the conclusion of a meeting, the end of an email thread, or when a ticket changes columns. The figure must be derived from records which cannot be altered afterwards, since a latency based on adjustable timestamps tells us nothing. You should report the median together with the number of suspensions. If an organization has no suspensions on file it should state “no suspensions,” not “zero.” An alarm which has never fired is not a fast alarm; it is merely one that has never been tested.
What counts as good? Looking at actual clinical incident data rather than relying on any survey standards: If the suspension lasts less than a day then there is someone on call. A suspension for a few days can be justified on the basis of having an on-call arrangement. Once it goes over a week, the organization has set up the suspension procedure but not the accountability loop, and a suspended model left in a state of suspension becomes a hazard in itself. Clinicians lose a tool without anyone explaining it to them; workarounds tend to become permanent; and there is a quiet pressure to simply re-enable it, which is how a safety measure ends up being just a formality. The most serious consequence is not a long delay; it is a model returning to service without any signed decision at all.
For your organization, the question is not just theoretical, and it can be settled in a single meeting. When was the last instance in which an AI system was automatically suspended and what was the time interval until a formal decision was reached? If your answer includes both a date and a number of hours, then you have a governance program. But if your answer is that there has never been an automatic suspension, then you should consider the next question by asking whether there are any thresholds that would cause such a suspension and whether they have ever been tested. And if the response is that suspensions occur informally, with someone deciding in a hallway to stop trusting the tool, then the program lacks both the necessary mechanism and the relevant record, and the binder full of policies describes a system that does not actually exist.
We have made the definition and rules for measurement publicly available under a license that allows any organization or vendor to adopt them, since a metric only disciplines a field when everyone calculates it in the same way. However, the definition is the simple part; the difficult part, as in my previous column, is that this figure cannot be produced the week before the survey. The organizations that will be in a position to state their decision latency two years from now are those for which monitoring has begun triggering and whose people have started signing off now.
Neel Chauhan is a physician-executive and the founder of The Healthcare AI Institute, an independent publisher and software maker for health care AI governance.
He is the author of fourteen operational titles on health care AI, covering governance, ambient documentation, revenue cycle, scheduling, and primary care operations. He also maintains the Institute’s open resources: the Transparency Registry of FDA AI clearances, the Agent Privileging Framework for governing AI that acts, and the working reference on The Joint Commission’s Responsible Use of AI in Healthcare certification.
The Institute accepts no vendor sponsorship, holds no vendor equity, and takes no referral fees. He shares updates on LinkedIn.
