The other day Atul Gawande tweeted the following:

Some still oppose WHO Safe Surgery Checklist because no RCT was done. Well, new RCT finds it cut complications 42% http://t.co/qWacE54NIW

— Atul Gawande (@Atul_Gawande) June 18, 2014

I am not against checklists. When I was a surgical chairman, I implemented and used one in both the operating room and the ICU. They do not add costs and may be helpful.

However, the randomized trial that Gawande referred to does not necessarily settle the issue about whether checklists really do reduce complications and deaths.

The paper, published online in Annals of Surgery, looked at 5,295 operations done in two Norwegian hospitals. The intervention was a 20-item checklist consisting of three critical steps: the sign in before anesthesia, the timeout before the operation began, and the sign out before the surgeon left the operating room. Using a stepped wedge cluster design, patients were randomized to control or the checklist.

Complications occurred in 19.9% of the control patients and 11.5% in those who got the checklist, a significant difference with p < 0.001.

A look at table 2 of the study finds that of 27 complications or groups of complications, 14 occurred in significantly fewer patients in the checklist group.

Of the significant 14, a few, such as cardiac or mechanical implant complications, could possibly have been prevented by the implementation of the checklist.

For most of the others, the relationship between the use of a checklist and a post-operative complication is tenuous. How could a checklist possibly prevent technical complications like bleeding requiring transfusion, surgical wound dehiscence, and unintended punctures or lacerations?

Here are a few more of the complications that occurred significantly less frequently in the checklist cohort—urinary tract infection, pneumonia, asthma, pleural effusion, dyspnea, and the nebulous categories of “complications after surgical and medical procedures” and “complications to surgery not classified.”

What item on a checklist prevents asthma, UTI or any of those on that list?

Embolism, sepsis, and surgical site infection, three complications one would expect a checklist to impact because of reminders to give prophylactic antibiotics and anticoagulation, did not occur at significantly lower rates in the checklist group.

Even the cardiac complication category is open to question because none of the 5 subcategories (cardiac arrest, arrhythmia, congestive heart failure, acute myocardial infarction) differed significantly between the two groups. Only when the 5 were combined did statistical significance emerge.

In the 300-bed community hospital, checklist use was associated with a significantly lower mortality rate than non-use, 0.2% vs. 1.9% respectively (p = 0.02), but no mortality difference was seen in the 1100-bed tertiary care hospital.

The tertiary care institution enrolled 3,811 patients while the 300-bed hospital contributed 1,083. If more patients had been in the latter group, the difference may have disappeared due to the principle of regression to the mean.

Despite the heightened vigilance associated with an ongoing research project, compliance with checklist use was only 73.4%.

Before you go off on me, I will remind you that I do not oppose checklists. Most things we do in medicine are not based on class 1 evidence.

Just don’t tell me that checklists have been proven to reduce complication rates or save lives.

“Skeptical Scalpel” is a surgeon blogs at his self-titled site, Skeptical Scalpel.