Within 48 hours of every job posting, hundreds of applications pour in. As the hiring manager, you want to fill the position within two months, or your supervisor might cut it out of your budget. After triaging the unqualified applicants using data-driven filters and artificial intelligence tools, you still have a few hundred left. At this point, many hiring managers invite the remaining candidates to take a situational judgment test (SJT). In medicine, the Objective Structured Clinical Examination (OSCE) is the most familiar example, and elements of the OSCE are part of the board and licensure exams. After sorting the results, hiring managers can refine their interview candidate shortlists.
The last decade has seen the adoption of SJTs in the medical/health professional application process with the Multiple Mini-Interview (MMI). A computer-based version called Casper was developed, and it helped schools that did not have resources to run in-person MMIs. Like other SJTs, Casper evaluated applicants on their interpersonal “soft skills” not captured in academic transcripts or exam scores, and it was widely adopted with medical schools in Canada.
However, U.S. medical schools wanted a test that reflected problematic situations they more frequently observed among their students. Adding an SJT to the MCAT proved untenable (compared to the UCAT exam in the United Kingdom), so the AAMC supported the development of an SJT to promote the holistic assessment of non-cognitive skills to shortlist candidates for interviews. In 2023, the PREview exam was fully implemented after many pilot studies showed its value.
This sounds like the Standardized Video Interview (SVI), which was being developed in parallel with residency programs, but SVI was dropped by 2020 because residency programs did not support it. However, individual residencies began to use Casper for their programs. Over the last few years, more residency candidates have taken Casper.
With the changing landscape of incorporating virtual components into the admissions process, the Health Professional Student Association (publishers of the Student Doctor Network) launched its SJT Experience Survey. In the 2023 survey, most respondents were premed and took the Casper or the PREview exam. The survey was conducted in the fall as the first offers of admission were extended (September to November) but before all classes were filled. Was there evidence that SJT results were being used as intended – an early screen to identify candidates for an interview among a pool of highly qualified candidates (shown by GPA and MCAT CARS scores)?
147 respondents gave their impressions of their SJT test-taking experiences with Casper and PREview. While most had strong academic records and performed well on the SJTs, they generally expressed skepticism of the relevance of these exams to their futures as health care professionals or students. Many admitted that they employed specific answering strategies that favored “acceptable” answers to the raters rather than revealed “authentic” responses. Since many candidates were interviewed when the survey closed, we could observe how programs selected candidates using their SJT results.
We found that schools were not using SJTs to screen candidates for interviews last cycle. Casper schools invited about 68% of our respondents (2.94 average quartile), but they did not pick respondents with significantly higher results (overall cohort 2.84 average quartile). PREview schools were less likely to invite respondents to interview (51% of all PREview respondents), and the average PREview score among interviewed candidates (6.12) was not significantly different from the overall average (5.90). Respondents with a first-quartile PREview score (1-3) were less likely to be interviewed (33%) than those with higher scores (50%), which could suggest early use of PREview scores for screening. Consequently, those who did well on SJTs found value in their scores, while those interviewed (despite low SJT scores) found them worthless in their possible acceptances.
Respondents embraced SJTs after they were selected for interviews. Half of the Casper-interviewed candidates believed that Casper had value in assessing candidates, and most of them disclosed fourth-quartile scores; in contrast, candidates with first- or second-quartile scores interviewed expressed more skepticism. Similarly, PREview-interviewed candidates with higher scores felt SJTs had value; interviewed candidates with lower scores were more skeptical. Regardless, our free-response comments clearly showed respondents had generally negative impressions of SJTs. Most wanted more transparency and evidence that SJTs fulfill their promise of selecting diverse candidates.
Acuity Insights and AAMC’s PREview teams have stated their SJTs should only be used to supplement information about the applicant and not to filter a large applicant pool. This position paradoxically contradicts why SJTs are used in other corporate and educational settings.
Because Canadian medical and dental schools have used Casper for a decade, data should be available to demonstrate how successful they have been in diversifying their student bodies. Has Casper reduced professionalism lapses or increased board exam performance (one should expect a correlation with OSCE performance)? Unfortunately, the Association of Faculties of Medicine in Canada (AFMC) has not progressed towards more diversity.
SJTs are now another part of the admissions and residency game. In the near future, applicants may pay a few hundred dollars to take these exams and perhaps up to a few thousand more for specialized coaching. Schools and residencies will not be afraid to use them for their stated purpose, and those with the resources and the right answers will be ready.
HPSA has made the Executive Summary available, with the full report on request. We would like to partner with any interested advocacy organizations to develop a similar survey for residency program applicants.
Emil Chuck is a health professional advisor.
