Skip to content
  • About
  • Contact
  • Contribute
  • Book
  • Careers
  • Podcast
  • Recommended
  • Speaking
  • All
  • Physician
  • Practice
  • Policy
  • Finance
  • Conditions
  • .edu
  • Patient
  • Meds
  • Tech
  • Social
  • Video
    • All
    • Physician
    • Practice
    • Policy
    • Finance
    • Conditions
    • .edu
    • Patient
    • Meds
    • Tech
    • Social
    • Video
    • About
    • Contact
    • Contribute
    • Book
    • Careers
    • Podcast
    • Recommended
    • Speaking

The ProPublica Surgeon Scorecard: When journalists become scientists

Peter Pronovost, MD, PhD
Policy
October 9, 2015
Share
Tweet
Share

Most of us would agree that there aren’t enough valid and meaningful health care quality measures to guide patients’ choices of hospitals and physicians. While the federal government has steadily expanded the number of publicly available measures on its Hospital Compare website, it still falls short of what many patients, payers, and providers would like. This is particularly true in the realm of outcomes such as infections and mortality rates, and in provider-level ratings.

Journalists and other ratings-making organizations have recently attempted to fill the measurement chasm left by policymakers and health care professionals. In July, nonprofit journalism organization ProPublica unveiled its Surgeon Scorecard, posting the “adjusted complication rates” for more than 16,000 physicians in eight inpatient procedures. The Scorecard’s release set off an intense debate within the health care community about the validity of the measure as well as the requirements of journalists when they function as scientists to create new measures. With the Surgeon Scorecard, ProPublica acted as judge and jury; they defined the measure, deemed it valid, and declared which surgeons were low quality. What assurances does the public have that such “vigilante” measures are scientifically sound? While ProPublica says its work was “guided by experts,” that review was informal.

Shortly after the Scorecard was issued, some detractors on social media called for it to undergo peer review, a process that is typical for government-issued measures. That review was delivered on Friday, when several researchers in health care quality measurement, including me, published a critique on the RAND Corporation website. Our conclusion: Patients should not consider the Scorecard a valid or reliable predictor of any individual surgeon’s outcomes.

Among several concerns raised, we pointed out that the adjusted complication rate, which was based mostly on readmissions, was not a true complication rate. The measure didn’t consider complications that occurred during a hospital admission and ignored many complications that are most meaningful to patients. For instance, erectile dysfunction is common after radical prostatectomy (more than 50 percent, according to some estimates), but it was not tracked in the ProPublica measure. We also found problems with the underlying data used by ProPublica: Some surgical cases were attributed to non-surgeons or to surgeons in the wrong specialty — a finding that suggests the existence of other errors that are harder to detect.

Developing and vetting a valid new quality measure can be hard, tedious and controversial. Yet that process unearths weaknesses, improves the final product, and ultimately makes the measure more useful to patients and physicians. No matter who creates a measure — the government, journalists or nonprofit groups — we all have a duty to ensure it receives the highest level of scrutiny before it’s issued, not after the fact. When journalists act as scientists, they should be held to the standards of scientists.

Peter Pronovost is an anesthesiologist and director, Armstrong Institute for Patient Safety and Quality.  He blogs at Voices for Safer Care, where this article originally appeared.

Image credit: Shutterstock.com

Prev

This is what patient-centered family medicine looks like

October 9, 2015 Kevin 7
…
Next

Why physicians face a leadership paradox

October 9, 2015 Kevin 5
…

Tagged as: Surgery

Post navigation

< Previous Post
This is what patient-centered family medicine looks like
Next Post >
Why physicians face a leadership paradox

ADVERTISEMENT

ADVERTISEMENT

ADVERTISEMENT

More by Peter Pronovost, MD, PhD

  • Explore the behavioral factors behind antibiotic misuse

    Peter Pronovost, MD, PhD
  • Revamp health regulations to reduce cost and improve patient safety

    Peter Pronovost, MD, PhD
  • How peer-to-peer review helps hospitals

    Peter Pronovost, MD, PhD

Related Posts

  • Paging the surgeon general: America needs you

    Linda Girgis, MD
  • Why creative endeavors are important for the future surgeon

    Thomas L. Amburn
  • To raise future scientists, address our innate curiosity  

    Jason J. Han, MD
  • Upset with the government’s pandemic response? Let’s elect more scientists and doctors.

    Shahdabul Faraz, MD
  • A trauma surgeon reflects on the Yale System, 20 years later

    Ara Feinstein, MD, MPH
  • Scientists predicted remdesivir’s success with a simulation. Here’s how.

    Cesar Padilla, MD

More in Policy

  • How locum tenens work helps physicians and APPs reclaim control

    Brian Sutter
  • Why Medicaid cuts should alarm every doctor

    Ilan Shapiro, MD
  • Why physician voices matter in the fight against anti-LGBTQ+ laws

    BJ Ferguson
  • The silent toll of ICE raids on U.S. patient care

    Carlin Lockwood
  • What Adam Smith would say about America’s for-profit health care

    M. Bennet Broner, PhD
  • The lab behind the lens: Equity begins with diagnosis

    Michael Misialek, MD
  • Most Popular

  • Past Week

    • Why Medicaid cuts should alarm every doctor

      Ilan Shapiro, MD | Policy
    • When the diagnosis is personal: What my mother’s Alzheimer’s taught me about healing

      Pearl Jones, MD | Conditions
    • 2 hours to decide my future: How the SOAP residency match traps future doctors

      Nicolette V. S. Sewall, MD, MPH | Education
    • Key strategies for smooth EHR transitions in health care

      Sandra Johnson | Tech
    • Reassessing the impact of CDC’s opioid guidelines on chronic pain care [PODCAST]

      The Podcast by KevinMD | Podcast
    • Why removing fluoride from water is a public health disaster

      Steven J. Katz, DDS | Conditions
  • Past 6 Months

    • Why tracking cognitive load could save doctors and patients

      Hiba Fatima Hamid | Education
    • What the world must learn from the life and death of Hind Rajab

      Saba Qaiser, RN | Conditions
    • The silent toll of ICE raids on U.S. patient care

      Carlin Lockwood | Policy
    • “Think twice, heal once”: Why medical decision-making needs a second opinion from your slower brain (and AI)

      Harvey Castro, MD, MBA | Tech
    • Why we fear being forgotten more than death itself

      Patrick Hudson, MD | Physician
    • Bureaucracy over care: How the U.S. health care system lost its way

      Kayvan Haddadan, MD | Physician
  • Recent Posts

    • How to survive a broken health care system without losing yourself [PODCAST]

      The Podcast by KevinMD | Podcast
    • Why some doctors age gracefully—and others grow bitter

      Patrick Hudson, MD | Physician
    • Why removing fluoride from water is a public health disaster

      Steven J. Katz, DDS | Conditions
    • What the research really says about infrared saunas

      Khushali Jhaveri, MD | Conditions
    • How the cycle of rage is affecting physicians—and how to break free

      Alexandra M.P. Brito, MD and Jennifer L. Hartwell, MD | Conditions
    • Why ADHD in adults is often missed—and why it matters [PODCAST]

      The Podcast by KevinMD | Podcast

Subscribe to KevinMD and never miss a story!

Get free updates delivered free to your inbox.


Find jobs at
Careers by KevinMD.com

Search thousands of physician, PA, NP, and CRNA jobs now.

Learn more

View 2 Comments >

Founded in 2004 by Kevin Pho, MD, KevinMD.com is the web’s leading platform where physicians, advanced practitioners, nurses, medical students, and patients share their insight and tell their stories.

Social

  • Like on Facebook
  • Follow on Twitter
  • Connect on Linkedin
  • Subscribe on Youtube
  • Instagram

ADVERTISEMENT

ADVERTISEMENT

ADVERTISEMENT

ADVERTISEMENT

  • Most Popular

  • Past Week

    • Why Medicaid cuts should alarm every doctor

      Ilan Shapiro, MD | Policy
    • When the diagnosis is personal: What my mother’s Alzheimer’s taught me about healing

      Pearl Jones, MD | Conditions
    • 2 hours to decide my future: How the SOAP residency match traps future doctors

      Nicolette V. S. Sewall, MD, MPH | Education
    • Key strategies for smooth EHR transitions in health care

      Sandra Johnson | Tech
    • Reassessing the impact of CDC’s opioid guidelines on chronic pain care [PODCAST]

      The Podcast by KevinMD | Podcast
    • Why removing fluoride from water is a public health disaster

      Steven J. Katz, DDS | Conditions
  • Past 6 Months

    • Why tracking cognitive load could save doctors and patients

      Hiba Fatima Hamid | Education
    • What the world must learn from the life and death of Hind Rajab

      Saba Qaiser, RN | Conditions
    • The silent toll of ICE raids on U.S. patient care

      Carlin Lockwood | Policy
    • “Think twice, heal once”: Why medical decision-making needs a second opinion from your slower brain (and AI)

      Harvey Castro, MD, MBA | Tech
    • Why we fear being forgotten more than death itself

      Patrick Hudson, MD | Physician
    • Bureaucracy over care: How the U.S. health care system lost its way

      Kayvan Haddadan, MD | Physician
  • Recent Posts

    • How to survive a broken health care system without losing yourself [PODCAST]

      The Podcast by KevinMD | Podcast
    • Why some doctors age gracefully—and others grow bitter

      Patrick Hudson, MD | Physician
    • Why removing fluoride from water is a public health disaster

      Steven J. Katz, DDS | Conditions
    • What the research really says about infrared saunas

      Khushali Jhaveri, MD | Conditions
    • How the cycle of rage is affecting physicians—and how to break free

      Alexandra M.P. Brito, MD and Jennifer L. Hartwell, MD | Conditions
    • Why ADHD in adults is often missed—and why it matters [PODCAST]

      The Podcast by KevinMD | Podcast

MedPage Today Professional

An Everyday Health Property Medpage Today
  • Terms of Use | Disclaimer
  • Privacy Policy
  • DMCA Policy
All Content © KevinMD, LLC
Site by Outthink Group

The ProPublica Surgeon Scorecard: When journalists become scientists
2 comments

Comments are moderated before they are published. Please read the comment policy.

Loading Comments...