Skip to content
  • About
  • Contact
  • Contribute
  • Book
  • Careers
  • Podcast
  • Recommended
  • Speaking
  • All
  • Physician
  • Practice
  • Policy
  • Finance
  • Conditions
  • .edu
  • Patient
  • Meds
  • Tech
  • Social
  • Video
    • All
    • Physician
    • Practice
    • Policy
    • Finance
    • Conditions
    • .edu
    • Patient
    • Meds
    • Tech
    • Social
    • Video
    • About
    • Contact
    • Contribute
    • Book
    • Careers
    • Podcast
    • Recommended
    • Speaking

How situational judgment tests help medical schools evaluate applicants

Emil Chuck, PhD
Education
March 13, 2024
Share
Tweet
Share

Within 48 hours of every job posting, hundreds of applications pour in. As the hiring manager, you want to fill the position within two months, or your supervisor might cut it out of your budget. After triaging the unqualified applicants using data-driven filters and artificial intelligence tools, you still have a few hundred left. At this point, many hiring managers invite the remaining candidates to take a situational judgment test (SJT). In medicine, the Objective Structured Clinical Examination (OSCE) is the most familiar example, and elements of the OSCE are part of the board and licensure exams. After sorting the results, hiring managers can refine their interview candidate shortlists.

The last decade has seen the adoption of SJTs in the medical/health professional application process with the Multiple Mini-Interview (MMI). A computer-based version called Casper was developed, and it helped schools that did not have resources to run in-person MMIs. Like other SJTs, Casper evaluated applicants on their interpersonal “soft skills” not captured in academic transcripts or exam scores, and it was widely adopted with medical schools in Canada.

However, U.S. medical schools wanted a test that reflected problematic situations they more frequently observed among their students. Adding an SJT to the MCAT proved untenable (compared to the UCAT exam in the United Kingdom), so the AAMC supported the development of an SJT to promote the holistic assessment of non-cognitive skills to shortlist candidates for interviews. In 2023, the PREview exam was fully implemented after many pilot studies showed its value.

This sounds like the Standardized Video Interview (SVI), which was being developed in parallel with residency programs, but SVI was dropped by 2020 because residency programs did not support it. However, individual residencies began to use Casper for their programs. Over the last few years, more residency candidates have taken Casper.

With the changing landscape of incorporating virtual components into the admissions process, the Health Professional Student Association (publishers of the Student Doctor Network) launched its SJT Experience Survey. In the 2023 survey, most respondents were premed and took the Casper or the PREview exam. The survey was conducted in the fall as the first offers of admission were extended (September to November) but before all classes were filled. Was there evidence that SJT results were being used as intended – an early screen to identify candidates for an interview among a pool of highly qualified candidates (shown by GPA and MCAT CARS scores)?

147 respondents gave their impressions of their SJT test-taking experiences with Casper and PREview. While most had strong academic records and performed well on the SJTs, they generally expressed skepticism of the relevance of these exams to their futures as health care professionals or students. Many admitted that they employed specific answering strategies that favored “acceptable” answers to the raters rather than revealed “authentic” responses. Since many candidates were interviewed when the survey closed, we could observe how programs selected candidates using their SJT results.

We found that schools were not using SJTs to screen candidates for interviews last cycle. Casper schools invited about 68% of our respondents (2.94 average quartile), but they did not pick respondents with significantly higher results (overall cohort 2.84 average quartile). PREview schools were less likely to invite respondents to interview (51% of all PREview respondents), and the average PREview score among interviewed candidates (6.12) was not significantly different from the overall average (5.90). Respondents with a first-quartile PREview score (1-3) were less likely to be interviewed (33%) than those with higher scores (50%), which could suggest early use of PREview scores for screening. Consequently, those who did well on SJTs found value in their scores, while those interviewed (despite low SJT scores) found them worthless in their possible acceptances.

Respondents embraced SJTs after they were selected for interviews. Half of the Casper-interviewed candidates believed that Casper had value in assessing candidates, and most of them disclosed fourth-quartile scores; in contrast, candidates with first- or second-quartile scores interviewed expressed more skepticism. Similarly, PREview-interviewed candidates with higher scores felt SJTs had value; interviewed candidates with lower scores were more skeptical. Regardless, our free-response comments clearly showed respondents had generally negative impressions of SJTs. Most wanted more transparency and evidence that SJTs fulfill their promise of selecting diverse candidates.

Acuity Insights and AAMC’s PREview teams have stated their SJTs should only be used to supplement information about the applicant and not to filter a large applicant pool. This position paradoxically contradicts why SJTs are used in other corporate and educational settings.

Because Canadian medical and dental schools have used Casper for a decade, data should be available to demonstrate how successful they have been in diversifying their student bodies. Has Casper reduced professionalism lapses or increased board exam performance (one should expect a correlation with OSCE performance)? Unfortunately, the Association of Faculties of Medicine in Canada (AFMC) has not progressed towards more diversity.

SJTs are now another part of the admissions and residency game. In the near future, applicants may pay a few hundred dollars to take these exams and perhaps up to a few thousand more for specialized coaching. Schools and residencies will not be afraid to use them for their stated purpose, and those with the resources and the right answers will be ready.

HPSA has made the Executive Summary available, with the full report on request. We would like to partner with any interested advocacy organizations to develop a similar survey for residency program applicants. 

Emil Chuck is a health professional advisor.

ADVERTISEMENT

Prev

Exploring the benefits of Dry January for older adults [PODCAST]

March 12, 2024 Kevin 0
…
Next

"No" can be a complete sentence

March 13, 2024 Kevin 0
…

Tagged as: Medical school

Post navigation

< Previous Post
Exploring the benefits of Dry January for older adults [PODCAST]
Next Post >
"No" can be a complete sentence

ADVERTISEMENT

More by Emil Chuck, PhD

  • Inside Out 2: How new emotions mirror medical student struggles

    Emil Chuck, PhD

Related Posts

  • Breaking the silence: the truth about mental health challenges among medical students and why medical schools must take action

    Erin Waldrop
  • The truth about Caribbean medical schools

    Jessica K. Willett, MD
  • Formalized mentorship as a requirement for medical schools

    Micaela Stevenson
  • We need more doctors. International medical schools can provide them.

    Richard Liebowitz, MD
  • The vital importance of climate change education in medical schools

    Helen Kim, MD
  • How the COVID-19 pandemic highlights the need for social media training in medical education 

    Oscar Chen, Sera Choi, and Clara Seong

More in Education

  • Why medical schools must ditch lectures and embrace active learning

    Arlen Meyers, MD, MBA
  • Why helping people means more than getting an MD

    Vaishali Jha
  • Residency match tips: Building mentorship, research, and community

    Simran Kaur, MD and Eva Shelton, MD
  • How I learned to stop worrying and love AI

    Rajeev Dutta
  • Why medical student debt is killing primary care in America

    Alexander Camp
  • Why the pre-med path is pushing future doctors to the brink

    Jordan Williamson, MEd
  • Most Popular

  • Past Week

    • Forced voicemail and diagnosis codes are endangering patient access to medications

      Arthur Lazarus, MD, MBA | Meds
    • How President Biden’s cognitive health shapes political and legal trust

      Muhamad Aly Rifai, MD | Conditions
    • The One Big Beautiful Bill and the fragile heart of rural health care

      Holland Haynie, MD | Policy
    • America’s ER crisis: Why the system is collapsing from within

      Kristen Cline, BSN, RN | Conditions
    • Why timing, not surgery, determines patient survival

      Michael Karch, MD | Conditions
    • How early meetings and after-hours events penalize physician-mothers

      Samira Jeimy, MD, PhD and Menaka Pai, MD | Physician
  • Past 6 Months

    • Forced voicemail and diagnosis codes are endangering patient access to medications

      Arthur Lazarus, MD, MBA | Meds
    • How President Biden’s cognitive health shapes political and legal trust

      Muhamad Aly Rifai, MD | Conditions
    • Why are medical students turning away from primary care? [PODCAST]

      The Podcast by KevinMD | Podcast
    • The One Big Beautiful Bill and the fragile heart of rural health care

      Holland Haynie, MD | Policy
    • Why “do no harm” might be harming modern medicine

      Sabooh S. Mubbashar, MD | Physician
    • Here’s what providers really need in a modern EHR

      Laura Kohlhagen, MD, MBA | Tech
  • Recent Posts

    • Why the heart of medicine is more than science

      Ryan Nadelson, MD | Physician
    • How Ukrainian doctors kept diabetes care alive during the war

      Dr. Daryna Bahriy | Physician
    • Why Grok 4 could be the next leap for HIPAA-compliant clinical AI

      Harvey Castro, MD, MBA | Tech
    • How women physicians can go from burnout to thriving

      Diane W. Shannon, MD, MPH | Physician
    • What a childhood stroke taught me about the future of neurosurgery and the promise of vagus nerve stimulation

      William J. Bannon IV | Conditions
    • Beyond burnout: Understanding the triangle of exhaustion [PODCAST]

      The Podcast by KevinMD | Podcast

Subscribe to KevinMD and never miss a story!

Get free updates delivered free to your inbox.


Find jobs at
Careers by KevinMD.com

Search thousands of physician, PA, NP, and CRNA jobs now.

Learn more

Leave a Comment

Founded in 2004 by Kevin Pho, MD, KevinMD.com is the web’s leading platform where physicians, advanced practitioners, nurses, medical students, and patients share their insight and tell their stories.

Social

  • Like on Facebook
  • Follow on Twitter
  • Connect on Linkedin
  • Subscribe on Youtube
  • Instagram

ADVERTISEMENT

  • Most Popular

  • Past Week

    • Forced voicemail and diagnosis codes are endangering patient access to medications

      Arthur Lazarus, MD, MBA | Meds
    • How President Biden’s cognitive health shapes political and legal trust

      Muhamad Aly Rifai, MD | Conditions
    • The One Big Beautiful Bill and the fragile heart of rural health care

      Holland Haynie, MD | Policy
    • America’s ER crisis: Why the system is collapsing from within

      Kristen Cline, BSN, RN | Conditions
    • Why timing, not surgery, determines patient survival

      Michael Karch, MD | Conditions
    • How early meetings and after-hours events penalize physician-mothers

      Samira Jeimy, MD, PhD and Menaka Pai, MD | Physician
  • Past 6 Months

    • Forced voicemail and diagnosis codes are endangering patient access to medications

      Arthur Lazarus, MD, MBA | Meds
    • How President Biden’s cognitive health shapes political and legal trust

      Muhamad Aly Rifai, MD | Conditions
    • Why are medical students turning away from primary care? [PODCAST]

      The Podcast by KevinMD | Podcast
    • The One Big Beautiful Bill and the fragile heart of rural health care

      Holland Haynie, MD | Policy
    • Why “do no harm” might be harming modern medicine

      Sabooh S. Mubbashar, MD | Physician
    • Here’s what providers really need in a modern EHR

      Laura Kohlhagen, MD, MBA | Tech
  • Recent Posts

    • Why the heart of medicine is more than science

      Ryan Nadelson, MD | Physician
    • How Ukrainian doctors kept diabetes care alive during the war

      Dr. Daryna Bahriy | Physician
    • Why Grok 4 could be the next leap for HIPAA-compliant clinical AI

      Harvey Castro, MD, MBA | Tech
    • How women physicians can go from burnout to thriving

      Diane W. Shannon, MD, MPH | Physician
    • What a childhood stroke taught me about the future of neurosurgery and the promise of vagus nerve stimulation

      William J. Bannon IV | Conditions
    • Beyond burnout: Understanding the triangle of exhaustion [PODCAST]

      The Podcast by KevinMD | Podcast

MedPage Today Professional

An Everyday Health Property Medpage Today
  • Terms of Use | Disclaimer
  • Privacy Policy
  • DMCA Policy
All Content © KevinMD, LLC
Site by Outthink Group

Leave a Comment

Comments are moderated before they are published. Please read the comment policy.

Loading Comments...