Recently, the Harvard Chan School of Public Health reported on the effect of surgical checklists in South Carolina. The press release was titled, “South Carolina hospitals see major drop in post-surgical deaths with nation’s first proven statewide Surgical Safety Checklist Program.”
The Health News Review, for which I review, grades coverage of research in the media. Based on their objective criteria, the Harvard press release would not score highly.
The title exudes certainty — “nation’s first proven.” The study, not being a randomized controlled trial (RCT), suggests that checklists are effective, but far from proves it. At least one study failed to show that surgical checklists improve outcomes.
The press release’s opening line is “South Carolina saw a 22 percent reduction in deaths.” It reports relative risk reduction (RRR). Reporting RRR is now considered a cardinal sin in health care journalism because RRR inflates therapeutic optimism by making the intervention sound more efficacious than it is.
A good press release gives the reader the whole picture of competing evidence, particularly where there’s controversy. The press release mentioned one other study which showed an even higher mortality benefit of surgical checklists.
The press release goes on to say:
Adoption of a safe surgery checklist has been demonstrated to reduce deaths in controlled research studies since 2009. But the ability to produce improved outcomes at large scale has remained questioned.
In other words, the science is settled. But there is no mention of the Canadian study which failed to show surgical checklists improve outcomes. The reader is lulled into a false sense of certainty about the benefits of surgical checklists.
If this was a press release for a statin or a device, which, ironically, subjects itself to greater methodological rigor, it would have been taken to task. Yet, no one batted an eyelid over the press release about surgical checklists. Why does a sensational press release about an inferior-quality study about surgical checklists not induce the same ire as a sensational press release about an RCT of a drug or device?
Future anthropologists might better answer this question. I’m not an anthropologist, let alone a future anthropologist. But I will speculate. In our rational age, we’re still indefatigably prejudiced, self-righteous creatures. Witnessing pharma make big bucks on false therapeutic optimism gets under our skin. Money corrupts. Nothing else quite corrupts to the same degree, we believe.
The proponents of surgical checklists don’t make money off checklists. That the proponents of checklists might advance their academic careers despite the therapeutic optimism of checklists doesn’t perturb. The nobility of their intent — they wish to make surgery safer — shields them from the harsh scrutiny reserved for drug and device industries.
It’s not readily apparent that the advocates of checklist might have the “this is my bright idea” bias. In this regard, the reaction of Atul Gawande, the author of Checklist Manifesto, who is self-evidently a true believer of checklists, to the Canadian study is instructive. He “wished the Ontario study was better.” He felt the study was underpowered and the implementation of checklists was weak.
When you don’t like the results of a study, the next step is to find weaknesses in the study design. The astute reader will note that I’ve done exactly what I’m accusing Gawande of doing. Guilty as charged. I have my biases. And there is no way to determine if one’s search for methodological weaknesses in a study is actuated by intrigue or bias. But my point isn’t whether surgical checklists work or not. My question is why do we not see the same skepticism for surgical checklists, and other policy measures, that we see for drugs and devices.
Of course, there are differences between checklists and drugs. Simple checklists cost nearly nothing to implement. If you’re wrong about the efficacy of a drug, that could be costly. If you’ve overestimated the efficacy of a surgical checklist, that’s no big deal.
But the differential standards, as justified as they may be, may lead to a deep mistrust of science and of dabblers in science. A casual observer, unfamiliar with epistemic nuances, may suspect that ideology is at play. The observer may see, in the sliding scale of methodological rigor, blatant double standards. The observer may lose faith in statistics seeing that it can be fine-tuned, like the thermostat of a shower, to prove and disprove what we wish proven or disproven.
When Anne Case and Angus Deaton showed that middle-class white men were not reaping the mortality benefits of other demographics, the demolition of their analysis was swift.
In a sense, this is good because science needs refutation. But would their study have inspired the same statistical scrutiny if it had shown that Hispanic or South Asian men were dying sooner than white men? Skepticism loses credibility if seen to be applied selectively. And if we can’t trust skeptics, who can we trust?
Statisticians can debunk or defend a study using their deep knowledge of statistics by selectively highlighting the strengths and weaknesses of a study — no study design is perfect. The degree to which the weaknesses of a study can be exposed is arbitrary — there’s no normative frame. Thus, statisticians have become the new lawyers. I wonder how long before we have a “right to a statistician.”
The ideologue with a deep knowledge of statistics is the most dangerous ideologue. The ideologue who claims God or history is on their side is easy to identify. The ideologue who claims to have science, which really means statistical technique, on their side is more difficult to identify and refute. Few have deep enough knowledge of statistics to call BS on the callers of BS. The result is that people mistrust science.
Science has become a substitute for morality. It has become a weapon to fight social injustice (whatever that term means), to reduce inequality, to mock incompatible world views. This is a blow for both science and morality. Historically, our species never resorted to science to do the right thing. Slavery was bad because it was morally bad. The Civil Rights Movement was right because it was morally right. The suffragettes were on the right side of history, not because an academic proved the net benefits of women voting in an election (p <0.0005), by analyzing a secondary database. The suffragettes were right because they were morally right. India’s freedom struggle was not initiated by a macroeconomic analysis, but because independence was morally right. If your moral intuitions can’t tell you that same-sex marriage should be permitted, I doubt any scientific analysis will convince you.
It may be morally right to be politically correct. Then we should say so, and keep science away from political correctness. Science loses most credibility when seen to selectively address the favored social injustices of the time. It is not fashionable in elite circles to be perturbed by the plight of the working class white man. In this circle, which I sometimes frequent, a physician seeping social justice from every pore, once said that it was imperative that we developed a vaccine for malaria, because the Africans have suffered enough from imperialism. He was right and wrong. He was right that a malaria vaccine will reduce the suffering of Africans tremendously, and that it was imperative that it be developed. But he was wrong. The vaccine is no more, or no less, important because of the history of the African continent. Alexander Fleming didn’t discover penicillin because he wanted to fight capitalists.
We may discourage reporting RRR for drugs and devices to curb therapeutic optimism and encourage reporting RRR for surgical checklists to encourage compliance. This is understandable, because how we frame a message, and reporting relative risk reduction is just a way of framing, depends on what we wish to achieve. But here lies another problem. Note, science is not informing us what to do. We’ve already decided what we want to do. Science is helping us implement, helping us engineer, helping us design, not a bridge, not a spacecraft, not a computer, but ourselves and society. This was once the dominion of religion. That this is now the dominion of science is ironic. But more troubling than irony is that science is no longer value-neutral. And when science ceases to be value-neutral, science ceases to be science.
Saurabh Jha is a radiologist and can be reached on Twitter @RogueRad. This article originally appeared in the Health Care Blog.
Image credit: Shutterstock.com
