Skip to content
  • About
  • Contact
  • Contribute
  • Book
  • Careers
  • Podcast
  • Recommended
  • Speaking
  • All
  • Physician
  • Practice
  • Policy
  • Finance
  • Conditions
  • .edu
  • Patient
  • Meds
  • Tech
  • Social
  • Video
    • All
    • Physician
    • Practice
    • Policy
    • Finance
    • Conditions
    • .edu
    • Patient
    • Meds
    • Tech
    • Social
    • Video
    • About
    • Contact
    • Contribute
    • Book
    • Careers
    • Podcast
    • Recommended
    • Speaking

Evidence-based medicine and the limitations of research

Elana Miller, MD
Conditions
December 23, 2013
Share
Tweet
Share

Before medical school I worked in a research lab investigating the relationship between stress and memory. As a research assistant, I dutifully administered memory test and collected saliva samples to test for cortisol levels.

My boss sent the data to her statistician for analysis, and was thrilled to find that despite the lack of connection between most of the variables of stress and memory studied, there was one positive finding — a connection between hippocampal volume (the part of the brain associated with memory) and life-long stress. I helped to write the article, and it was published in a major medical journal.

It wasn’t until years later that I realized my boss was engaging in an extremely common, but scientifically misleading, practice called data mining. Instead of having one specific hypothesis (i.e. that high cortisol levels would correlate with memory loss as measured on one specific test), she looked at numerous variables associated with stress and memory with the hope that some positive result could be dug out from the heaps of data.

The problem with this approach is that if you test enough variables, you will, by chance alone, find a result that meets the somewhat arbitrary criteria for “statistical significance” that will allow your work to be published as a positive finding. For example (and this is an oversimplified example), if you flip a quarter enough times, eventually you may gets heads 10 times in a row, but this doesn’t mean you’ve got a special quarter.

The pressure in the scientific community to publish positive results is enormous. After all, negative results don’t often advance careers or bring fame and fortune to the person who publishes them. It is not uncommon for researchers to data mine, to do statistical acrobatics, and even in more extreme cases to flat-out create fake data in order to get published (the New York Times reported earlier this year that Diederik Stapel, a Dutch social scientist and academic star, had risen to fame by faking experiments that were published in major journals for over a decade).

As Stapel admitted to the reporter writing the story, “My behavior shows that science is not holy.”

In my article on the placebo effect, I referenced Irving Kirsch, who published a paper, and then a popular book, arguing that antidepressants act only as placebos in the vast majority of people. His work seemed well-researched and well-referenced, and he even contacted the FDA to get results of unpublished trials done by pharmaceutical companies (pharmaceutical companies generally avoid publishing results that show their medications don’t work — shocking, I know).

But then I watched a lecture by two psychiatrists and researchers who went through Kirsch’s work and pointed out a few significant problems with it. Kirsch argued that antidepressants don’t work for people with mild or moderate depression. However, he misclassified mild, moderate, and severe symptoms as mild, moderate, and severe depression, when, by definition, a person must have severe symptoms in order to be diagnosed with depression at all (if you only have a few severe symptoms, you would be diagnosed with mild depression).

When these psychiatrists reanalyzed Kirsch’s data using the correct classification, they found that antidepressants worked better than placebo for people with both moderate and severe depression, and that it was only in cases of mild depression where antidepressants didn’t beat the placebo.

Sound complicated? Well, the TV show 60 Minutes agreed. After Kirsch appeared on the show telling the world that antidepressants were no better than placebo, these two psychiatrists spent over an hour on the phone with the producers explaining the issues with his research.

Apparently they listened attentively, and agreed that the issue was more complicated that Kirsch made it seem, but decided not to have them on the show to counter. Whereas statistics is a complex science, news outlets like to push out stories that are simple and sensationalized.

So there are problems with the way research is conducted and reported, but what’s a well-intentioned healthcare practitioner to do? It doesn’t make sense to completely ignore a large body of scientific research that can help us practice medicine in a safer and more effective way. Instead of ignoring scientific research, or preaching it from the rooftops as if it were gospel, I propose a middle way.

First, let’s be honest that much of what we currently believe to be true in medicine will be disproven at some point. I remember the first time I ever heard that something I had learned as dogma in medical school was totally wrong; it was only six months after I graduated.

ADVERTISEMENT

This reality is perhaps more true in psychiatry than in any other field, where we deal with an incredibly complex organ (the brain), and an even more complex system that is connected to the brain in a way we don’t fully understand (the mind).

Second, knowing that our knowledge base is ever evolving, let’s not be afraid to use some common sense. Do I really need a research study showing that being an empathetic and supportive physician improves outcomes to believe that it’s true? Or if a patient tells me a natural supplement helps them, should I tell them, “No, you’re wrong” just because there’s not yet a double-blinded, placebo-controlled trial on it?

Which brings me to my next point — the concept of “evidenced-informed” rather than “evidence-based” medicine, which I first heard proposed at a conference on integrative medicine. Integrative medicine is a field that takes a holistic view of the patient, and combines both traditional, allopathic treatments with complementary and alternative approaches, such as acupuncture, homeopathy, herbal medicine, etc.

Skeptical allopathic doctors will often say that there’s “no proof” these alternative therapies work, but I’d argue that’s not the whole story. Instead, when evaluating the evidence base of a treatment (whether traditional or alternative), let’s consider its 1) safety, 2) tolerability, 3) cost, 4) efficacy, and 5) convenience.

Examining a treatment like chemotherapy, for example, which is potentially dangerous, expensive, and inconvenient, we should require a lot of research showing that it is effective before recommending it. But take a treatment like a multivitamin, which is safe, cheap, without side effects, and convenient. Do we really need a dozen positive studies before recommending it to our patients?

Lastly, let’s be skeptical of people or organizations with obvious agendas, no matter how much research they quote. With thousands of articles published a day, it’s easy to find a few dozen that support your agenda, whatever it may be.

Recently I read a book by the journalist Robert Whitaker called Anatomy of an Epidemic: Magic Bullets, Psychiatric Drugs, and the Astonishing Rise of Mental Illness in America. He argued, persuasively, that many psychiatric medications, most notably antipsychotics, are more harmful than helpful, and I naively assumed he was presenting an accurate and complete picture of the relevant literature.

At a few points I found myself rolling my eyes at his biased and selective stories of patients who were miraculously cured when they stopped taking their medications, but it wasn’t until I got to the chapter on lithium and bipolar disorder that I had to put down the book in disgust.

It was here that it became abundantly clear that Whitaker had selectively cited articles that supported his agenda, while leaving out dozens of well-known and well-done studies that presented a different, and more nuanced, point of view. Sigh. I guess sensationalism must sell more books than nuance.

At the opposite end of the spectrum, it’s probably wise to be skeptical of research presented and published by pharmaceutical companies. Earlier this year I was at the annual conference of the American Psychiatric Association, and during one of the breaks found myself wandering to the booth of Otsuka America Pharmaceuticals because they were offering free ice cream (yes, I’m easy to manipulate).

As I waited in line for my ice cream, I tried to avoid making eye contact with any of the reps, but one started chatting with me anyway. I told him I didn’t have much experience working with reps because because they’re not allowed on campus at the academic medial center where I work.

He looked at me, aghast, and asked, “But how do you get education, then?”

I was speechless. Umm … residency? Conferences? Lectures? Articles? Books? Supervisors? Mentors? I think I’m fine on the education front, thanks.

So in sum, instead of practicing “evidence-based medicine” as if the evidence were dogma, let’s use research to move the practice of medicine forward while also remembering its limitations, including biases against publishing negative results and the agendas of the people doing or reporting the literature.

Let’s consider practicing “evidence-informed medicine” instead, where we consider factors like safety, tolerability, and cost when deciding how much evidence we need to see before recommending a treatment.

And lastly, let’s stay humble, respect how complex the body and brain are, and remember that there’s much that we still don’t know.

Elana Miller is a psychiatrist who blogs at Zen Psychiatry.

Prev

Stop asking radiologists to rule out disease

December 23, 2013 Kevin 18
…
Next

After grieving and healing: Transformation

December 23, 2013 Kevin 2
…

Tagged as: Psychiatry

Post navigation

< Previous Post
Stop asking radiologists to rule out disease
Next Post >
After grieving and healing: Transformation

ADVERTISEMENT

ADVERTISEMENT

ADVERTISEMENT

More by Elana Miller, MD

  • So I am still here. I live to fight another day.

    Elana Miller, MD
  • These stunning photos take you on a physician’s cancer journey

    Elana Miller, MD
  • a desk with keyboard and ipad with the kevinmd logo

    8 ways to better support someone dealing with cancer

    Elana Miller, MD

More in Conditions

  • Measles is back: Why vaccination is more vital than ever

    American College of Physicians
  • Hope is the lifeline: a deeper look into transplant care

    Judith Eguzoikpe, MD, MPH
  • From hospital bed to harsh truths: a writer’s unexpected journey

    Raymond Abbott
  • Bird flu’s deadly return: Are we flying blind into the next pandemic?

    Tista S. Ghosh, MD, MPH
  • “The medical board doesn’t know I exist. That’s the point.”

    Jenny Shields, PhD
  • When moisturizers trigger airport bomb alarms

    Eva M. Shelton, MD and Janmesh Patel
  • Most Popular

  • Past Week

    • The silent toll of ICE raids on U.S. patient care

      Carlin Lockwood | Policy
    • Why recovery after illness demands dignity, not suspicion

      Trisza Leann Ray, DO | Physician
    • Addressing the physician shortage: How AI can help, not replace

      Amelia Mercado | Tech
    • Why medical students are trading empathy for publications

      Vijay Rajput, MD | Education
    • Why does rifaximin cost 95 percent more in the U.S. than in Asia?

      Jai Kumar, MD, Brian Nohomovich, DO, PhD and Leonid Shamban, DO | Meds
    • Why physician voices matter in the fight against anti-LGBTQ+ laws

      BJ Ferguson | Policy
  • Past 6 Months

    • What’s driving medical students away from primary care?

      ​​Vineeth Amba, MPH, Archita Goyal, and Wayne Altman, MD | Education
    • Make cognitive testing as routine as a blood pressure check

      Joshua Baker and James Jackson, PsyD | Conditions
    • The hidden bias in how we treat chronic pain

      Richard A. Lawhern, PhD | Meds
    • A faster path to becoming a doctor is possible—here’s how

      Ankit Jain | Education
    • Residency as rehearsal: the new pediatric hospitalist fellowship requirement scam

      Anonymous | Physician
    • The broken health care system doesn’t have to break you

      Jessie Mahoney, MD | Physician
  • Recent Posts

    • Why physician voices matter in the fight against anti-LGBTQ+ laws

      BJ Ferguson | Policy
    • From burnout to balance: a lesson in self-care for future doctors

      Seetha Aribindi | Education
    • How conflicts of interest are eroding trust in U.S. health agencies [PODCAST]

      The Podcast by KevinMD | Podcast
    • Why young doctors in South Korea feel broken before they even begin

      Anonymous | Education
    • Measles is back: Why vaccination is more vital than ever

      American College of Physicians | Conditions
    • When errors of nature are treated as medical negligence

      Howard Smith, MD | Physician

Subscribe to KevinMD and never miss a story!

Get free updates delivered free to your inbox.


Find jobs at
Careers by KevinMD.com

Search thousands of physician, PA, NP, and CRNA jobs now.

Learn more

View 4 Comments >

Founded in 2004 by Kevin Pho, MD, KevinMD.com is the web’s leading platform where physicians, advanced practitioners, nurses, medical students, and patients share their insight and tell their stories.

Social

  • Like on Facebook
  • Follow on Twitter
  • Connect on Linkedin
  • Subscribe on Youtube
  • Instagram

ADVERTISEMENT

ADVERTISEMENT

ADVERTISEMENT

ADVERTISEMENT

  • Most Popular

  • Past Week

    • The silent toll of ICE raids on U.S. patient care

      Carlin Lockwood | Policy
    • Why recovery after illness demands dignity, not suspicion

      Trisza Leann Ray, DO | Physician
    • Addressing the physician shortage: How AI can help, not replace

      Amelia Mercado | Tech
    • Why medical students are trading empathy for publications

      Vijay Rajput, MD | Education
    • Why does rifaximin cost 95 percent more in the U.S. than in Asia?

      Jai Kumar, MD, Brian Nohomovich, DO, PhD and Leonid Shamban, DO | Meds
    • Why physician voices matter in the fight against anti-LGBTQ+ laws

      BJ Ferguson | Policy
  • Past 6 Months

    • What’s driving medical students away from primary care?

      ​​Vineeth Amba, MPH, Archita Goyal, and Wayne Altman, MD | Education
    • Make cognitive testing as routine as a blood pressure check

      Joshua Baker and James Jackson, PsyD | Conditions
    • The hidden bias in how we treat chronic pain

      Richard A. Lawhern, PhD | Meds
    • A faster path to becoming a doctor is possible—here’s how

      Ankit Jain | Education
    • Residency as rehearsal: the new pediatric hospitalist fellowship requirement scam

      Anonymous | Physician
    • The broken health care system doesn’t have to break you

      Jessie Mahoney, MD | Physician
  • Recent Posts

    • Why physician voices matter in the fight against anti-LGBTQ+ laws

      BJ Ferguson | Policy
    • From burnout to balance: a lesson in self-care for future doctors

      Seetha Aribindi | Education
    • How conflicts of interest are eroding trust in U.S. health agencies [PODCAST]

      The Podcast by KevinMD | Podcast
    • Why young doctors in South Korea feel broken before they even begin

      Anonymous | Education
    • Measles is back: Why vaccination is more vital than ever

      American College of Physicians | Conditions
    • When errors of nature are treated as medical negligence

      Howard Smith, MD | Physician

MedPage Today Professional

An Everyday Health Property Medpage Today
  • Terms of Use | Disclaimer
  • Privacy Policy
  • DMCA Policy
All Content © KevinMD, LLC
Site by Outthink Group

Evidence-based medicine and the limitations of research
4 comments

Comments are moderated before they are published. Please read the comment policy.

Loading Comments...