The risks of publically reported surgical outcomes

“Some data is better than no data at all.” Do you believe that?

I heard it frequently when the infamous ProPublica’s Surgeon Scorecard first appeared three years ago. Back then I blogged about it saying “To me, bad data is worse than no data at all.”

A recent study in BJU International confirmed my thoughts about this type of publicly posted data and identified a previously unreported issue. The paper attempted to determine whether the public was able to accurately interpret statistics used in the Surgeon Scorecard. It turns out they were not very good at it.

Investigators from the department of urology at the University of Minnesota surveyed 343 people who attended the Minnesota State Fair in 2016. Those who took the survey had a median age of 48, were 60% female, 80% white, and 60% college educated. Their median annual income was $26,550 with an interquartile range of $22,882 to $32,587.

The authors showed individuals the figure below on a tablet computer with the accompanying statement “This graph shows the individual surgeons’ complication rates after 28-35 cases. Surgeons A, B and C raw complication rates are A = 1/35 or 2.9%, B = 1/34 or 3.8% and C = 1/28 or 3.6%.”

In case you aren’t sure, understand the complication rates for these three surgeons are not significantly different due to the small numbers of cases and complications.

The most surprising finding of the study was although the surgeons’ complication rates were clearly stated above the figure, just 15.2% of the participants could correctly identify surgeon C’s complication rate. The participants thought the average complication rate for surgeon C was 25% (range 3.6% to 50%). Regarding surgeon B, they were better at estimating the complication rate, but still only 34.9% got it right.

The subjects were asked multiple-choice questions related to the surgeons’ complication rates. When asked to choose a surgeon for a hypothetical procedure, 192 (56%) picked surgeon A, 30 (8.7%) picked B, and 19 (5.5%) selected C; 102 (29.7%) said they didn’t have enough information to decide.

Here’s the new wrinkle on the potential harm of misinterpreting data. The subjects were then told that their insurance would only pay if they used surgeon C, and if they wanted to use one of the other surgeons, they would have to pay out of their own pockets. Almost two-thirds said they would pay an average of $5,754 in order to have their surgery done by surgeon A or B.

Those willing to switch were significantly poorer, had a significantly higher incidence of a history of cancer, and misinterpreted the complication rates significantly more often.

What this means is that the people who could least afford to switch surgeons were the most likely to do so.

Bottom line: People may misinterpret published data on surgical complication rates which could result in financial harm to them.

“Skeptical Scalpel” is a surgeon who blogs at his self-titled site, Skeptical Scalpel.