Skip to content
  • About
  • Contact
  • Contribute
  • Book
  • Careers
  • Podcast
  • Recommended
  • Speaking
  • All
  • Physician
  • Practice
  • Policy
  • Finance
  • Conditions
  • .edu
  • Patient
  • Meds
  • Tech
  • Social
  • Video
    • All
    • Physician
    • Practice
    • Policy
    • Finance
    • Conditions
    • .edu
    • Patient
    • Meds
    • Tech
    • Social
    • Video
    • About
    • Contact
    • Contribute
    • Book
    • Careers
    • Podcast
    • Recommended
    • Speaking

How situational judgment tests help medical schools evaluate applicants

Emil Chuck, PhD
Education
March 13, 2024
Share
Tweet
Share

Within 48 hours of every job posting, hundreds of applications pour in. As the hiring manager, you want to fill the position within two months, or your supervisor might cut it out of your budget. After triaging the unqualified applicants using data-driven filters and artificial intelligence tools, you still have a few hundred left. At this point, many hiring managers invite the remaining candidates to take a situational judgment test (SJT). In medicine, the Objective Structured Clinical Examination (OSCE) is the most familiar example, and elements of the OSCE are part of the board and licensure exams. After sorting the results, hiring managers can refine their interview candidate shortlists.

The last decade has seen the adoption of SJTs in the medical/health professional application process with the Multiple Mini-Interview (MMI). A computer-based version called Casper was developed, and it helped schools that did not have resources to run in-person MMIs. Like other SJTs, Casper evaluated applicants on their interpersonal “soft skills” not captured in academic transcripts or exam scores, and it was widely adopted with medical schools in Canada.

However, U.S. medical schools wanted a test that reflected problematic situations they more frequently observed among their students. Adding an SJT to the MCAT proved untenable (compared to the UCAT exam in the United Kingdom), so the AAMC supported the development of an SJT to promote the holistic assessment of non-cognitive skills to shortlist candidates for interviews. In 2023, the PREview exam was fully implemented after many pilot studies showed its value.

This sounds like the Standardized Video Interview (SVI), which was being developed in parallel with residency programs, but SVI was dropped by 2020 because residency programs did not support it. However, individual residencies began to use Casper for their programs. Over the last few years, more residency candidates have taken Casper.

With the changing landscape of incorporating virtual components into the admissions process, the Health Professional Student Association (publishers of the Student Doctor Network) launched its SJT Experience Survey. In the 2023 survey, most respondents were premed and took the Casper or the PREview exam. The survey was conducted in the fall as the first offers of admission were extended (September to November) but before all classes were filled. Was there evidence that SJT results were being used as intended – an early screen to identify candidates for an interview among a pool of highly qualified candidates (shown by GPA and MCAT CARS scores)?

147 respondents gave their impressions of their SJT test-taking experiences with Casper and PREview. While most had strong academic records and performed well on the SJTs, they generally expressed skepticism of the relevance of these exams to their futures as health care professionals or students. Many admitted that they employed specific answering strategies that favored “acceptable” answers to the raters rather than revealed “authentic” responses. Since many candidates were interviewed when the survey closed, we could observe how programs selected candidates using their SJT results.

We found that schools were not using SJTs to screen candidates for interviews last cycle. Casper schools invited about 68% of our respondents (2.94 average quartile), but they did not pick respondents with significantly higher results (overall cohort 2.84 average quartile). PREview schools were less likely to invite respondents to interview (51% of all PREview respondents), and the average PREview score among interviewed candidates (6.12) was not significantly different from the overall average (5.90). Respondents with a first-quartile PREview score (1-3) were less likely to be interviewed (33%) than those with higher scores (50%), which could suggest early use of PREview scores for screening. Consequently, those who did well on SJTs found value in their scores, while those interviewed (despite low SJT scores) found them worthless in their possible acceptances.

Respondents embraced SJTs after they were selected for interviews. Half of the Casper-interviewed candidates believed that Casper had value in assessing candidates, and most of them disclosed fourth-quartile scores; in contrast, candidates with first- or second-quartile scores interviewed expressed more skepticism. Similarly, PREview-interviewed candidates with higher scores felt SJTs had value; interviewed candidates with lower scores were more skeptical. Regardless, our free-response comments clearly showed respondents had generally negative impressions of SJTs. Most wanted more transparency and evidence that SJTs fulfill their promise of selecting diverse candidates.

Acuity Insights and AAMC’s PREview teams have stated their SJTs should only be used to supplement information about the applicant and not to filter a large applicant pool. This position paradoxically contradicts why SJTs are used in other corporate and educational settings.

Because Canadian medical and dental schools have used Casper for a decade, data should be available to demonstrate how successful they have been in diversifying their student bodies. Has Casper reduced professionalism lapses or increased board exam performance (one should expect a correlation with OSCE performance)? Unfortunately, the Association of Faculties of Medicine in Canada (AFMC) has not progressed towards more diversity.

SJTs are now another part of the admissions and residency game. In the near future, applicants may pay a few hundred dollars to take these exams and perhaps up to a few thousand more for specialized coaching. Schools and residencies will not be afraid to use them for their stated purpose, and those with the resources and the right answers will be ready.

HPSA has made the Executive Summary available, with the full report on request. We would like to partner with any interested advocacy organizations to develop a similar survey for residency program applicants. 

Emil Chuck is a health professional advisor.

ADVERTISEMENT

Prev

Exploring the benefits of Dry January for older adults [PODCAST]

March 12, 2024 Kevin 0
…
Next

"No" can be a complete sentence

March 13, 2024 Kevin 0
…

Tagged as: Medical school

Post navigation

< Previous Post
Exploring the benefits of Dry January for older adults [PODCAST]
Next Post >
"No" can be a complete sentence

ADVERTISEMENT

More by Emil Chuck, PhD

  • Inside Out 2: How new emotions mirror medical student struggles

    Emil Chuck, PhD

Related Posts

  • Breaking the silence: the truth about mental health challenges among medical students and why medical schools must take action

    Erin Waldrop
  • The truth about Caribbean medical schools

    Jessica K. Willett, MD
  • Formalized mentorship as a requirement for medical schools

    Micaela Stevenson
  • We need more doctors. International medical schools can provide them.

    Richard Liebowitz, MD
  • The vital importance of climate change education in medical schools

    Helen Kim, MD
  • How the COVID-19 pandemic highlights the need for social media training in medical education 

    Oscar Chen, Sera Choi, and Clara Seong

More in Education

  • Why clinical research is a powerful path for unmatched IMGs

    Dr. Khutaija Noor
  • Dear July intern: It’s normal to feel clueless—here’s what matters

    Tomi Mitchell, MD
  • Why medical schools must ditch lectures and embrace active learning

    Arlen Meyers, MD, MBA
  • Why helping people means more than getting an MD

    Vaishali Jha
  • Residency match tips: Building mentorship, research, and community

    Simran Kaur, MD and Eva Shelton, MD
  • How I learned to stop worrying and love AI

    Rajeev Dutta
  • Most Popular

  • Past Week

    • Who gets to be well in America: Immigrant health is on the line

      Joshua Vasquez, MD | Policy
    • Why specialist pain clinics and addiction treatment services require strong primary care

      Olumuyiwa Bamgbade, MD | Conditions
    • Harassment and overreach are driving physicians to quit

      Olumuyiwa Bamgbade, MD | Physician
    • Why peer support can save lives in high-pressure medical careers

      Maire Daugharty, MD | Conditions
    • When a medical office sublease turns into a legal nightmare

      Ralph Messo, DO | Physician
    • Addressing menstrual health inequities in adolescents

      Callia Georgoulis | Conditions
  • Past 6 Months

    • Forced voicemail and diagnosis codes are endangering patient access to medications

      Arthur Lazarus, MD, MBA | Meds
    • How President Biden’s cognitive health shapes political and legal trust

      Muhamad Aly Rifai, MD | Conditions
    • Why are medical students turning away from primary care? [PODCAST]

      The Podcast by KevinMD | Podcast
    • The One Big Beautiful Bill and the fragile heart of rural health care

      Holland Haynie, MD | Policy
    • Who gets to be well in America: Immigrant health is on the line

      Joshua Vasquez, MD | Policy
    • Why “do no harm” might be harming modern medicine

      Sabooh S. Mubbashar, MD | Physician
  • Recent Posts

    • The shocking risk every smart student faces when applying to medical school

      Curtis G. Graham, MD | Physician
    • Clinical ghosts and why they haunt our exam rooms

      Kara Wada, MD | Conditions
    • High blood pressure’s hidden impact on kidney health in older adults

      Edmond Kubi Appiah, MPH | Conditions
    • Deep transcranial magnetic stimulation for depression [PODCAST]

      The Podcast by KevinMD | Podcast
    • How declining MMR vaccination rates put future generations at risk

      Ambika Sharma, Onyi Oligbo, and Katrina Green, MD | Conditions
    • The physician who turned burnout into a mission for change

      Jessie Mahoney, MD | Physician

Subscribe to KevinMD and never miss a story!

Get free updates delivered free to your inbox.


Find jobs at
Careers by KevinMD.com

Search thousands of physician, PA, NP, and CRNA jobs now.

Learn more

Leave a Comment

Founded in 2004 by Kevin Pho, MD, KevinMD.com is the web’s leading platform where physicians, advanced practitioners, nurses, medical students, and patients share their insight and tell their stories.

Social

  • Like on Facebook
  • Follow on Twitter
  • Connect on Linkedin
  • Subscribe on Youtube
  • Instagram

ADVERTISEMENT

  • Most Popular

  • Past Week

    • Who gets to be well in America: Immigrant health is on the line

      Joshua Vasquez, MD | Policy
    • Why specialist pain clinics and addiction treatment services require strong primary care

      Olumuyiwa Bamgbade, MD | Conditions
    • Harassment and overreach are driving physicians to quit

      Olumuyiwa Bamgbade, MD | Physician
    • Why peer support can save lives in high-pressure medical careers

      Maire Daugharty, MD | Conditions
    • When a medical office sublease turns into a legal nightmare

      Ralph Messo, DO | Physician
    • Addressing menstrual health inequities in adolescents

      Callia Georgoulis | Conditions
  • Past 6 Months

    • Forced voicemail and diagnosis codes are endangering patient access to medications

      Arthur Lazarus, MD, MBA | Meds
    • How President Biden’s cognitive health shapes political and legal trust

      Muhamad Aly Rifai, MD | Conditions
    • Why are medical students turning away from primary care? [PODCAST]

      The Podcast by KevinMD | Podcast
    • The One Big Beautiful Bill and the fragile heart of rural health care

      Holland Haynie, MD | Policy
    • Who gets to be well in America: Immigrant health is on the line

      Joshua Vasquez, MD | Policy
    • Why “do no harm” might be harming modern medicine

      Sabooh S. Mubbashar, MD | Physician
  • Recent Posts

    • The shocking risk every smart student faces when applying to medical school

      Curtis G. Graham, MD | Physician
    • Clinical ghosts and why they haunt our exam rooms

      Kara Wada, MD | Conditions
    • High blood pressure’s hidden impact on kidney health in older adults

      Edmond Kubi Appiah, MPH | Conditions
    • Deep transcranial magnetic stimulation for depression [PODCAST]

      The Podcast by KevinMD | Podcast
    • How declining MMR vaccination rates put future generations at risk

      Ambika Sharma, Onyi Oligbo, and Katrina Green, MD | Conditions
    • The physician who turned burnout into a mission for change

      Jessie Mahoney, MD | Physician

MedPage Today Professional

An Everyday Health Property Medpage Today
  • Terms of Use | Disclaimer
  • Privacy Policy
  • DMCA Policy
All Content © KevinMD, LLC
Site by Outthink Group

Leave a Comment

Comments are moderated before they are published. Please read the comment policy.

Loading Comments...