Subscribe to The Podcast by KevinMD. Watch on YouTube. Catch up on old episodes!

Join us as we delve into the evolving landscape of health professional admissions processes, focusing on the use of situational judgment tests (SJTs) like Casper and PREview. Our guest, Emil Chuck, a health professional advisor, sheds light on the effectiveness and implications of SJTs in identifying candidates for interviews amidst a competitive applicant pool. We discuss candidate perceptions, the integration of SJTs into admissions protocols, and the potential impact on diversity and equity within health care education.

Emil Chuck is a health professional advisor.

He discusses the KevinMD article, “How situational judgment tests help medical schools evaluate applicants.”

Our presenting sponsor is Nuance, a Microsoft company.

Together, Microsoft and Nuance are leveraging their rich digital technology and advanced AI capabilities to tackle some of health care’s biggest challenges. AI-driven technology promises to revolutionize patient and provider experiences with clinical documentation that writes itself.

The Nuance Dragon Ambient eXperience, or DAX for short, is a voice-enabled solution that automatically captures patient encounters securely and accurately at the point of care. DAX Copilot combines proven conversational and ambient AI with the most advanced generative AI in a mobile application that integrates directly with your existing workflows.

Physicians who use DAX have reported a 50 percent decrease in documentation time and a 70 percent reduction in feelings of burnout, and 85 percent of patients say their physician is more personable and conversational.

Discover AI-powered clinical documentation that writes itself. Visit https://nuance.com/daxinaction to see a 12-minute DAX Copilot demo.

VISIT SPONSOR → https://nuance.com/daxinaction

SUBSCRIBE TO THE PODCAST → https://kevinmd.com/podcast

RECOMMENDED BY KEVINMD → https://kevinmd.com/recommended

GET CME FOR THIS EPISODE → https://earnc.me/U98Fsc

Powered by CMEfy. 

Transcript

Kevin Pho: Hi, and welcome to the show. Subscribe at KevinMD.com/podcast, and get CME for this episode by clicking on the CME link in the show notes. Today we welcome Emil Chuck. He’s a health professional advisor. Today’s KevinMD article is “How situational judgment tests help medical schools evaluate applicants.” Emil, welcome to the show.

Emil Chuck: Kevin, thank you for having me, and I’m looking forward to this conversation.

Kevin Pho: We’ll talk about your article in a little bit. First off, briefly share your story and journey.

Emil Chuck: So I have a PhD in cell biology from Case Western Reserve University, after having an engineering background at Duke University. I spent a number of years in the research world doing postdocs before basically making the turn towards academic administration, where I was the first university-wide pre-health advisor at George Mason University, then director of admissions at the dental school at Case Western Reserve for about nine years or so, and then, especially during the pandemic, just by coincidence became director of admissions at Rosalind Franklin University, overseeing roughly 20 or so of the health professions programs there.

About three years ago I left that position for this very interesting position as director of advising services for the Health Professional Student Association, which is well known for publishing the Student Doctor Network, now entering basically its 25th anniversary year. So we’re doing a lot of work, and a lot of the work that I’ve currently been doing is leveraging some of my research interests and looking at what the applicant experience really has been, not just for medicine but also for many, many different professions.

Kevin Pho: All right, so as many of my listeners know, applying to the health professions, especially medical school and dental school, is a long and arduous process. I have a freshman in college on the premedical track as well, so she is going through that journey.

Let’s talk about this KevinMD article about situational judgment tests. So tell us more about this article for those who didn’t get a chance to read it.

Emil Chuck: So a little bit of background. I get a lot of insights from many of the faculty, many clinicians, or even people in the general business world, who talk a lot about how, generationally, a lot of our students don’t seem to understand what professionalism is all about.

And so a number of different efforts along many, many years have been pursued, especially in higher education, to gauge what we will term the soft skills, the power skills, those skills that are professional in nature.

And so what I try to do with this article, at least with a little bit of background, is highlight the two main players right now in health professions admissions when it comes to evaluating these soft skills. So we’re not talking about the content skills and knowledge that you know for the MCAT, the DAT, and so forth, but these are soft skills about interprofessionalism, ethics, integrity, working in teams, and sort of having some nuance and understanding of cultural awareness and cultural competency.

So these tests have been developed over at least the last 10 years, and they really started bubbling up and becoming more prominent in the last couple of years, mainly driven by the AAMC’s situational judgment test. But these other tests have also been in play for a number of years, and I just wanted to make sure we highlight them, because very, very little is known about how students and how applicants really feel about how effective these assessments really are and how it plays a role in the admissions process.

Kevin Pho: So do premedical students have to take these situational judgment tests alongside the MCAT? Are they required by all medical schools today?

Emil Chuck: At this time, for medical school admissions, these tests are not required. They’re only required by a small handful of schools. AAMC, because they rolled it out for implementation for the first year last year, they’re trying to get some buy-in from everybody. So right now the number of required schools is relatively small, less than a dozen at this point, but that may change in a couple of years as more and more buy-in and more and more research comes in for the AAMC PREview exam.

Other medical schools are using the Casper exam, and if you happen to be in Canada, the Casper exam is a required part of your application experience for pretty much most of the medical and dental schools up there.

And actually one of the things that I found was very surprising is that for many other health professions they’re using this Casper situational judgment test. The physician assistant programs, there’s roughly 50 programs using and asking their applicants to take the Casper exam as part of their admissions screening process.

So this is a very underrated story that has been going on for at least a decade, and I just wanted to make sure that with the AAMC now promoting a situational judgment test, that more and more people become more aware of these exams. It’s not absolutely required yet, but needless to say a lot of the emphasis on these professional skills and competencies is making its way, and who knows, we’ll see if all the schools will wind up adopting it for AMCAS.

Kevin Pho: And are these tests administered by the schools themselves, or is this something that the applicant has to take independently, like an MCAT?

Emil Chuck: As it currently stands, these situational judgment tests are administered by a third-party vendor. So AAMC has partnered with an outside vendor. You have to take them and schedule them as if you were going to take the MCAT. Casper does the same thing, you have a specific website that you register for an individual test date, and you go and take the test according to that specific date and that specific time.

I’m not going to go into school specific variations of that type of test yet, that’s a topic of another article that I’m certainly working on. But at this particular point, these are meant to basically complement the required exams that you would normally take, such as the MCAT and the DAT.

Kevin Pho: So take us into one of these tests. You mentioned a Casper test. So what is the experience of a typical test taker? What kind of scenarios do they find themselves in, what kind of questions do they get asked, how long is the test? Just give us a sense of what it’s like to take this test.

Emil Chuck: So one of the things this article really tries to do, or our report tries to do, is it actually highlights the fact that these two tests are different. They seem to be constructed differently, and thus they reward different things.

So the Casper exam, which has been around for a little bit longer, is an open-ended test. You actually have to type in, or answer into a webcam, your responses to various prompted questions that are given to you. And many of the scenarios that you get are, I would basically describe, as general life experiences. So say you have to return an item to a store, but the store has this particular policy about requiring a receipt, how would you react to that type of situation? And those sorts of things do exist. It really depends on the day in terms of what scenarios you get. But over the span of about 90 minutes, that’s what Casper examinees experience. A number of exam questions or situations require verbal responses while others require typed responses.

The AAMC’s test though, they take a different direction. So similar to the MCAT, they prefer multiple choice type exams, and it is meant to be valid over a number of different years. So even if you take an exam for say 2022, it could be valid for up to, well, depending on the school, say 2025 or 2026, unlike the Casper exam, which you take it then and there and it’s only valid within the 12 months that you take the exam.

The AAMC PREview exam is specifically designed to really gauge the participants who take the exam about, in my opinion, academic culture and academic medical life. What is expected of you in terms of scenarios in an academic medical setting, where you have to compete against various priorities in terms of your commitments and other things, and your integrity, your teamwork skills, and so forth.

And so a lot of the scenarios that we see on their practice tests, for example, really focus a lot on how well individuals will succeed and understand what the game plan is, what the so-called curriculum and the expectations are, should you become a health professional or medical student at the AMCAS schools. And I should put in parenthesis, or an asterisk in there, there are a few DO schools that are also adopting PREview as part of their process, but I don’t know if they’ve required it yet, I think they’re still doing research on it.

Kevin Pho: Now, what’s the data regarding the efficacy of these tests? Have they been really shown to choose a better applicant pool?

Emil Chuck: So that’s the, I guess, $10 million question among all of the people who are creating these types of tests.

Casper fortunately has been around for at least a decade, and so they have a number of different peer-reviewed publications where they really do partner with some of their participating programs and demonstrate a lot of the benefit of individuals that have taken these exams for its institutional outcomes. So in some cases you might see a smaller number of problems when it comes to professionalism citations, you might see people performing better on OSCE exams. Those sorts of things are currently in the literature when it comes to the Casper exam.

And we’re hopeful that in the future, because PREview really hasn’t been fully implemented except for a number of three or four years, there’s a smaller number of publications that really show a similar benefit, but they have been published and they’re making their way through the literature right now. So there’s a couple of papers that really have identified, during pilot studies, how well it contributes to those issues related to professionalism, how it pertains to board passage, we’re hoping we’ll hear more about that soon, how it relates to professionalism citations while you’re in school or even beyond. Hopefully that’s forthcoming.

But there’s a bunch of people over at AAMC that are really trying to collaborate with many of the member institutions to come up with that research, and hopefully that’ll be data that will be rolling out in the near future.

Kevin Pho: As you know, it’s already so rigorous for American medical students to apply, right? You have to have your GPA, you have to have the MCAT, you have to have your shadowing hours, the research hours and clinical hours, all to apply to medical school. And now you have these situational judgment tests. Are we going to have a world where we’re going to have a Casper or a PREview coach or tutor that these premedical students are going to have to prepare for these tests?

Emil Chuck: Well, that world exists now. So believe it or not, if you go and you see what resources are available for Casper outside of the test maker website, which is Acuity Insights, they do have some resources that are freely available for people to become acclimated to the tests. But there are many guides already, even for us at Student Doctor Network and HPSA. We’ve had a number of people who are outside admissions consultants and test taking companies who outline what their general strategy is when it comes to answering questions on Casper.

So I think that world already currently exists, and I would certainly say for any standardized test, which PREview is designed to be, a standardized test, the more that we know about it, I think the more people will figure out what the examiners are really looking for, and find ways to really properly prepare for those exams as well.

But I also am not sure whether that type of coaching is really going to be effective, because it’s not really asking for a correct answer per se. In a lot of cases it is the way you approach the answer, and so how decision making skills and alternate outcomes are assessed really are the focus of these exams, and not so much a yes or no type answer. So it’s a little bit different in terms of coaching compared to if you had to take the MCAT or the DAT.

Kevin Pho: It seems to be like being coached to make decisions so they can do well in these tests. Doesn’t it defeat the purpose of situational judgment tests and what they were designed for?

Emil Chuck: I think, one, it is very ironic, I would agree, that why are we testing situational judgment, because I will agree many of these skills are early on developed when you were in grade school. And there aren’t significant differences depending on your socioeconomic status in terms of really understanding the right way, the wrong way. That should be independent of that.

And so we’ve known through Casper that those types of scoring differentials that we normally will see on the MCAT based on socioeconomic status are much smaller for the Casper exam, as one example. And I would expect that for PREview it might also be a little bit smaller, because we’re not asking about things that may have a longer tail or have some influence from history in terms of right or wrong.

But one of the things that is important is, I think if we do this way too much, there is always a sense of, we’re all just going to check the boxes, we’re going to answer to what the people want us to answer, and whether that’s an authentic representation of who we are or where we grew up, I think it’s a little bit more of an interesting question.

So you think that it’s kind of wrong to be testing this, but nowadays we have this everywhere. If you wound up applying to Google or Facebook or some of the big tech companies who get thousands of job applications a year, many companies are employing their own version of a situational judgment test to also sort of create their short list of candidates that they want to pursue further for interviews. So it’s not that foreign of a world, it’s just foreign for most of us in the United States. And situational judgment tests have been part of the admissions process around the world, especially in the UK, and Casper is currently being used, I think, no, the UCAT is being used in Australia and New Zealand in addition to the UK.

So I think we want people who are very strong in decision making, who are very considerate and thoughtful in a professional way, and that they know that’s what’s going to be expected of them for years and years to come as professionals.

Kevin Pho: So you have the pulse of the premedical community, of course, on Student Doctor Network. What is the general reaction to the situational judgment test among the premedical community?

Emil Chuck: Well, obviously, no one wants to have more hoops to jump through. And so I think a lot of the concern about the situational judgment tests, even though we have a lot of information about them, is the fact that very little is known about how these tests are really being used in admissions decisions.

Even though there’s been over 10 years of experience with Casper, none of the medical schools in Canada really have given any insight on how they use those scores to select candidates for their medical or dental schools. And that lack, or a mismatch, of information really upsets many of the applicants, because they’re very skeptical that any of these tests really mean anything.

Even if you publish all this other work that any of these organizations are going to do, it doesn’t boil down to, well, what do I need to do in order to show I’m really prepared for a health professional program, in the same way that we can say a specific range of an MCAT or GPA or DAT score makes you a stronger candidate for a certain number of schools. That data is still not very forthcoming, and I think that contributes to the sense of skepticism and lack of value for these types of tests among all the people who are applying to schools. And it even frustrates those who are their pre-health advisors.

I could tell you that, as a person who was involved in admissions, I would know certain information about a person who took Casper. I would know their percentile as an applicant in terms of how well they scored relative to their entire pool. However, the applicants themselves, they only know sort of their range, what band or what quartile they happen to be in. And it’s probably just as fair, do you really need to know you were 76th percentile versus 85th percentile on a situational judgment test, and what does that mean?

So I think we have to be absolutely sure we adopt appropriate language that we can allay some of these concerns, that people are just going to be cut off or screened out just because they didn’t make the top 25th percentile, 50th percentile of an applicant pool. But what does that really mean?

In general, what we don’t know is what the other health professions programs are doing and how they’re using these tests. And one of the things that is very interesting is that very few of us who have been in admissions who use these tests ever talk to our peers to find out, is this really a good practice, a best practice, that we could use these tests, and how does it affect the types of students that you wind up getting for your program? So there’s a lot that’s still out there, and having been one of the unusual people who’ve been on both sides of understanding that data disparity has been very interesting, and is something that I try to address to our pre-health community.

Kevin Pho: Now, for those premedical students listening to you here, what kind of advice do you have for them regarding a situational judgment test, in terms of the resources that they should spend preparing for this test? Is it something they really need to worry about? Should they hire coaches and tutors for this test? How much time should they spend preparing for a situational judgment test in relation to everything else that they have to do to apply to medical school?

Emil Chuck: So I think as originally designed, these SJTs are similar to how the SJT is used for the UCAT, the UK version of their MCAT. Situational judgment is like the last section, it’s very, very brief, and it’s something that you take sort of at the end of your entire testing experience. And I want to encourage our students to similarly think that way.

Until there is something very, very vastly different, nothing is going to upset the main pillars of your application, being your academic performance and what that history looks like in terms of your MCAT score. So most people who have answered our survey scored really, really well, have very good academic credentials, and most of them in this case tend to be in the higher percentiles, the top percentile range, the top quartile for Casper, the top scores for PREview. So they all tend to do pretty well, and none of them adopted any extra effort to get a coach or do anything more than just look at what resources are available from each of the test taking sites, maybe review some strategies, do a little bit of practice, some of the resources we have available for SDN to really tell people about, here’s how you can write some of your responses, or what people suggest for Casper. And I think for most people that’s all you really need to worry about, except for the technology.

And so really the amount of time you should spend is on making sure the technology you have is appropriate, that you have continuous internet, that you’re not in a place where you’re going to be easily distracted, that you can read the material. And that if you need accommodations, you request them very early on. If you have qualified for a fee assistance program, you register and make sure that that fee assistance program gets you reduced or free registration to these tests.

I think a lot of the work really comes up front in terms of preparing yourself for the test, and then making sure that as you’re going through the test, understanding that these are very standardized questions, don’t read too heavily into nuances or anything else like that, but answer forthrightly relative to who you’re trying to answer to and who the assessors are. I think if you do that, generally most people are going to do pretty well in these exams.

Kevin Pho: Are there any anecdotal stories where an applicant had impeccable academic credentials and did not get into medical school because of a situational judgment test? I know that the whole process is kind of opaque, but any anecdotal stories that you heard from Student Doctor Network?

Emil Chuck: Well, I don’t think I’ve heard any stories, whether through Student Doctor Network or from my peers in the academic pre-health advising or admissions communities, telling me anything that someone has that scenario or situation.

Most admissions processes are embracing a process where we’re looking at holistic reviews. So even if someone has a very low or relatively low SJT score, does that necessarily mean they’re out of the running for getting into a health professions program? It’s hard to say, and it’s hard to really narrow down those types of stories. And until we have data that says the contrary, it’s hard to basically say, oh, just because you got a low Casper score or a low PREview score, you’re out.

The data that we have from our report suggests that at this point a lot of schools have a lot of tolerance, and there are many people who got interviews in spite of having low SJT scores. And so I think that is a little bit more of a better connection in terms of whether these SJTs are really making an effect on the admissions process at that time point.

Other admissions people talk about using SJTs more as a so-called plus factor, an additional sort of benefit if we’re going to do tiebreakers or whatever else. That’s what many other people are saying that they’re going to do, and so that type of systematic data to sort of demonstrate that is much harder to deduce, unless the schools and all the programs themselves really start making an effort to show their work and really say, here’s the range of individuals, just like we have the range of GPA, the range of MCAT scores, here’s the range and maybe even the relative proportion of people who scored on Casper, who scored on PREview. And do our very best to say, this is the range, this is not meaning that this is the average that you’ve got to hit or do better than, or anything else that we normally associate, language wise, when we look at these same charts for DAT, MCAT, and GPA.

Kevin Pho: So I’m going to ask you to look into your crystal ball now. Where do you see situational judgment tests heading in the future? Do you anticipate a model similar to Canada where the majority of medical schools are going to require this test? What do you see based on your experience?

Emil Chuck: Well, one thing that gives me a little bit of an indication of where SJTs are going is seeing how a similar situation came about for residencies. And actually there are many residency programs that are using Casper as an SJT as part of the residency selection process, and that’s something that is still very much underreported.

And I would love to have people collaborate with me, especially some organizations, program directors, whatever associations of them, to try to figure that piece out, to allay the fears of the student deans that are trying to advise their students who have to take the Casper, such as the urology match. So is there something going on with that?

But I think in the future, it’s just really hard to get the buy-in from so many different medical schools. I do think that since AAMC put so much energy and so much money into developing the PREview exam, I think once we get much more data, I do think that there are going to be more people that will buy in and actually have more students take that exam. AAMC reported that 20,000 students took their PREview exam in their first full year of implementation this past last year, and that’s just about over a third of the overall applicant pool for MDs.

And so I think we’re going to go into a world where obviously that type of situational judgment testing and looking at people pre-professionally will be part of an admissions process that’s going to be holistic. What I think needs to be the rubber hitting the road is exactly what are the outcomes that we’re looking for that are going to be important. How does this affect overall diversity, if in and of itself or in combination with other factors and processes? How is this going to affect individual choices to go into certain professions, especially to work with underserved communities? And that’s of course something that we at HPSA are very interested in.

So we’re looking at a lot of things that are happening. But I’ll definitely say this, that for Canada, because of the smaller number of schools that are there, they’re missing some very interesting opportunities to really address this question from a Casper standpoint and showing how effective it is in addressing some of the problems they have in Canada with regards to diversity and regarding choices for specialties.

And I think in other episodes you probably have talked to many people who are experts in the Canadian health care system, and there’s still some significant problems despite 10 years of having Casper being part of their admissions process, and their selection process for residency, I may add. So we’re not really sure exactly. You have to justify your work, and I’m hoping that in a few years we get more data that really points us in a direction where we see that these SJTs are really going to be valuable.

Kevin Pho: And my final question, Emil. Tell us some of your take-home messages that you want to leave with the KevinMD audience.

Emil Chuck: So I think one of the things that we need to remember is that SJTs I think are really here to stay. These virtual interviewing platforms that really look at pre-professional activities and pre-professional attitudes among our candidates, I think it’s going to be something that we’ve all wanted and we’ve all craved.

We know the people, and many of your listeners and viewers are probably in the category of those that didn’t really do well as academics but had the heart and had the desire and had the resilience to be the types of physicians and health care workers they want to be. And I think to that extent, we’re hoping that the message out there is, don’t lose hope if you are one of those people with a strong heart and maybe had a very rocky road in an academic type setting. We’re hoping that this is going to be part of an overall assessment of individual applicants, that you are part of our health care system and we want you there too.

So I think, as much as we possibly can, keep your eyes on the ball, but understand the intentions of what these tests may actually mean in terms of shaping a much more humanistically based system.

Kevin Pho: Emil, very interesting. Thank you so much for sharing your perspective and insight, and thanks again for coming on the show.

Emil Chuck: Thank you, Kevin, I appreciate it.