A medical emergency call is never just a phone call. On the other side of the line, there may be chest pain, shortness of breath, trauma, fear, confusion, or a family member trying to describe symptoms without medical vocabulary. The first information is often incomplete. The situation may change within minutes. A decision has to be made quickly, but it cannot be made blindly.

In São Paulo, Brazil, where I work as a physician in emergency medical regulation, this is part of the daily reality of the Mobile Emergency Care Service (SAMU), the public mobile emergency care system. Calls are initially received and classified through a structured process, but the medical decision is not left to the system alone. A physician regulator reviews the request, evaluates the presumed severity, may confirm or change the priority, defines the most appropriate available resource, and remains accountable for the technical decision.

This matters because a protocol can organize information, but it cannot fully understand the patient.

A structured triage system is essential. It helps separate immediate life-threatening emergencies from lower-priority requests. It helps allocate ambulances, advanced life support units, and hospital destinations in a more rational way. It creates a common language for risk. But the physician regulator adds something that no checklist can fully replace: clinical judgment under uncertainty.

The same lesson applies to artificial intelligence in health care. As AI tools become more common in claims review, fraud detection, prior authorization, and payment decisions, we are being asked to trust systems that can process large amounts of data faster than any human team. These tools can be useful. They may identify unusual billing patterns, flag claims that deserve closer review, and help overwhelmed audit teams decide where to look first.

But a flagged claim is not a final diagnosis. An anomaly score is not proof of fraud. A model output is not a complete clinical story.

In health insurance claims review, the risk is not only that AI may miss suspicious patterns. The risk is also that AI may overcall them. A legitimate claim may look unusual because the patient was complex, the procedure was rare, the documentation was imperfect, or the local billing pattern differed from the data used to train the model. If an algorithmic flag automatically leads to denial, delay, or accusation, the consequences reach far beyond administration. They affect physicians, patients, access to care, and trust in the health system.

This is why I believe health care should borrow a principle from emergency medical regulation: Automation can support triage, but accountable human review must remain central to the final decision.

In emergency regulation, structured systems help organize urgency. But when a call is reviewed by a physician, the system gains a safety layer. The physician can ask whether the classification makes sense. Is the patient deteriorating? Does the initial code underestimate the risk? Is the closest destination appropriate, or does this patient need a specialized center? Should the plan change based on new information from the field team? That is not resistance to systems. That is governance.

AI in claims review needs the same type of governance. It should help prioritize cases for audit, not replace the audit itself. It should show why a claim was flagged, what variables contributed to the alert, and how confident the model is. It should be validated locally, monitored continuously, and reviewed by professionals who understand both clinical care and administrative rules.

A physician auditor reviewing an AI-flagged claim can ask questions the model cannot answer on its own. Was the service clinically justified? Was the documentation consistent with the diagnosis? Could the unusual pattern reflect a legitimate high-risk population? Is there evidence of abuse, or simply complexity? Should this case lead to denial, education, further documentation, or no action? These questions are not technical details. They are the difference between responsible oversight and blind automation.

The future of AI in health care should not be a battle between physicians and machines. It should be a redesign of decision-making. Machines are good at scanning large datasets. Physicians are trained to interpret uncertainty, context, risk, and consequences. Health care organizations need both.

But they also need clear accountability. If AI flags a claim and no one can explain why, the system is not ready for high-stakes use. If a payment decision affects a patient or physician and no human professional can review it, the process is unsafe. If leaders evaluate AI only by accuracy while ignoring false positives, false negatives, appeal burden, and trust, they are measuring the wrong thing.

Emergency medicine teaches us that speed matters, but speed without judgment can be dangerous. The same is true in health care administration. AI can help us see patterns earlier. It can help audit teams focus their attention. It can support sustainability in systems under financial pressure. But it should remain a tool for clinical governance, not a substitute for it.

Before health systems scale AI in claims review, they should ask a simple question: Who is the physician regulator of this decision? If no one can answer, the system is not ready.

Carolline Chagas Moreira is a physician in Brazil.