Listen and followApple PodcastsSpotifyYouTubeMore apps · Browse all episodes
A resident read a student’s personal statement and asked why she didn’t just use ChatGPT. The comment landed like a wound. In this episode, certified coach and professor Kathleen Muldoon returns to unpack what a residency personal statement is actually for, and why an algorithm can’t do it for you. This episode is based on her article “Why ChatGPT can’t write your residency personal statement,” published on KevinMD. You will hear why the product was never the essay but the self-knowledge that comes from writing it, how AI tends to average out the distinctive voice and sharp edges that make an applicant feel human, and where a tool like ChatGPT can still help without taking over the whole process. She also weighs whether AI detection tools unfairly flag applicants whose first language isn’t English. Hear why she says the real question was never whether AI wrote it, but whether the person still comes through.
This episode is brought to you by ModMed.
Welcome to your new AI-Powered Practice from ModMed. We’re transforming specialty care by embedding AI Assistants across your entire workflow. Our all-in-one platform of EHR, patient engagement, practice management, and RCM helps reduce repetitive work while keeping you firmly in control.
Trained on de-identified data from nearly a billion patient encounters, this isn’t just smarter software. It’s a new way of working for specialty medicine, more efficient and more connected to the patient experience.
Start building your AI-Powered Practice at modmed.com.
VISIT SPONSOR → https://www.modmed.com/
SUBSCRIBE TO THE PODCAST → https://www.kevinmd.com/podcast
Transcript
Kevin Pho: Hi, and welcome to the show. Subscribe at KevinMD.com/podcast. Today we welcome back Kathleen Muldoon, certified coach and professor. Today’s KevinMD article is “Why ChatGPT can’t write your residency personal statement.” Kathleen, welcome back to the show.
Kathleen Muldoon: Thanks for having me.
Kevin Pho: All right, so I think last time we talked, we talked about the residency personal statement. This one is another story related to that. So tell me what happened here.
Kathleen Muldoon: This piece came to me because a student who was getting ready to submit her residency application had been working with me in one of my personal statement writing workshops. She had written this absolutely gorgeous piece, where she developed a very strong narrative arc about her journey into palliative care medicine and made meaning of all of the observations she had collected during her time on rotations. It was a really wonderful, strong personal statement.
She read it, or offered it, to a resident for some feedback on one of the rotations she was on, and his response was basically, “It’s good, but why didn’t you just use ChatGPT to write it?”
Real physician voices, twice a week
Free, and one click to unsubscribe.
Kevin Pho: The resident said that?
Kathleen Muldoon: The resident said that. And so she came back to me to help process that comment, because what she heard was, “You wasted your time writing this,” and that it was not very efficient to do the writing.
In this age where we are all struggling with how AI is forcing its way into our processes and into medical education, this piece came out of that, and out of really trying to understand what the point is of the personal essay part of the residency application process. In my opinion, spending time reflecting on who you are and how to make meaning out of the moments that stood out to you in parts of your training is never a waste of time, and it is also not something that can be outsourced to an algorithm.
Kevin Pho: There’s a certain amount of irony here, because that resident, of course, has completed medical school and got accepted to a residency, yet that resident felt that generative AI could replace the reflection needed for a personal statement. This is someone who is, of course, applying to residency.
Kathleen Muldoon: Yes. The irony is rich. And that statement could also belie everyone grappling with how do we use this and how do we not use it. There could be a lot that is lost in the context of where that comment came from, but the way that it landed with the student, I think, also speaks to something that comes out in the process of reflecting on who you are and where you want to land in medicine in terms of the next step of your training.
Someone who notices, someone who wants to be their true, authentic self, those are not necessarily things that are promoted during medical education. And so the way that it landed with this person who is about to lay everything out on the table, and we have talked about this before, to an algorithm that makes a match that determines how you are going to continue your career in medicine, can feel devastating. So I thought it was worth reflecting on.
Kevin Pho: So as much as you can, tell us about some of the conversations you had with this prospective resident, and how you helped her process that comment. It’s almost dismissive. Why didn’t you just use ChatGPT to write this, after, like you said, she laid out all her cards on the table? How did you process that with her?
Kathleen Muldoon: Well, I think there is a lot of value in talking about things in workshop formats, and in not only giving voice to your thoughts and fears and opinions on things, but also hearing back how it lands. Because then it gives you the opportunity to say, “Does this sound like me? Is what I am saying, reflected back to me by what other people say, how I want to be perceived?” We cannot always control how the words that we use land, because everyone is bringing their own narrative and experiences to it.
So what we spent some time doing was unpacking what she had learned through the process of writing. In these workshops that I run, we read out parts of our pieces, and I ask the question, “Does this sound like you? Are you proud of this? Is the meaning that other people are reflecting back to you about that moment changing anything about your interpretation? Is it not the interpretation that you want people to have?” All of these kinds of moments, which take time and take reflection, are things we work through. And, was that a valuable part of your process?
I think it is easy to think about the personal statement as a writing assignment, like a box you have to check when you are submitting your application, because you do. But in my mind, the product isn’t the essay. The product is understanding the person you are in the process of writing that, and the doctor that you want to be.
And so, just asking her in that moment, in this coaching moment that we had, is that time that you regret? Is that time that you learned something, and how are you going to bring that forward? Working through that with her, we were able to come to the understanding that she is very proud of this piece of work. It is a true reflection of who she is, and it also answers the question that will come up when you are in the interview process of the application cycle: Is this someone I want on my team at 2:00 a.m.? Is this someone that I can spend the next few years of training with? That is a nuance AI does not necessarily pick up on for you. Only you can make that meaning.
Kevin Pho: Absolutely. So the journey of writing the personal statement itself, of course, is so much more valuable than the actual product, like you said, that comes out. Now, you have talked to a lot of medical students as they have gone through, of course, your workshop. Just give us a sampling of some of the surprising things that they may have learned about themselves during that whole personal statement process.
Kathleen Muldoon: Yeah. I usually run the workshops in four-week increments. You don’t come in with a polished draft. We are not editing out the thoughts and commas and em dashes and all the things that have come up as markers of something that AI wrote. Instead it is, what is a memorable moment for you? And even taking several. OK, now write five. Now write a little more detail about that. What makes it memorable for you?
And going through a process of, well, what is the defining similarity between these moments that stand out to you? It is not just the pattern of noticing, but it is also going back to what you noticed and understanding, wow, what is really coming up for me here is that team making is really something that is important to me, and why I want to be part of a really strong emergency medicine residency program.
A second thing that is always surprising to students is that they think they need to be really impressive on the page. And I will say, well, those things will come out on your CV, and all of the impressive things like that. What I think the personal statement gives you space to do is those smaller moments: Can you make meaning of them? When you are the student, because these are just students, sitting in the room with somebody whose treatment plan is not going as they wanted or as their family wanted, but you are the person who sits there and does have the time that you may not always have to sit with the patient and hear what matters to them, then what you are experiencing is your core value in wanting to develop that ability to ask the questions that allow the patient to tell you their deepest fears, their most vulnerable moments, or why something is so hard for them. And it may not be just the diagnosis. It might be a lot of other things.
And so what is surprising to them, actually, I think, is that it doesn’t have to be the most impressive, glossy, polished language moments. It is, can you make meaning of those things that stick out in your memory, which is that self-reflective process.
Kevin Pho: Now, I am going to assume that you have read many, many, many personal statements. How confident are you to detect that something, a paragraph or a sentence, is written by AI versus not AI?
Kathleen Muldoon: I am pretty confident. And it is not just the polished language and some of the grammatical cues that tend to have been picked up by AI, but it is a lack, especially in conversation with the students, of their ability to flesh out those moments and why they matter, or how they relate to their journey, how they relate to the kind of practice that they want to end up in.
And so, because these large language models are trained on vast amounts of data, and their purpose is to predict the next best word, it comes out polished. I love Margaret Atwood, a Canadian author, and she calls it the great blanding, because it tends to make you average. What gets lost is the distinctive voice, the odd moments, the sharp edges. Those things tend to get polished out in a statement that is written by generative AI, because it is not something that an algorithm can make sense of. It is not something that follows the predictive pattern.
And the large language models are getting better, but I have noticed that I can still sense when it is just averaging out those parts that I think students are afraid to include because they seem a little bit out there.
But I do stand firm, especially after talking to many residency directors and the people who are reading these statements, especially now that so many are getting averaged out, that the ones that stand out are the ones that feel human, that feel real, and that the students who are applying can speak about honestly and authentically in conversation.
Kevin Pho: Is there a role for AI at all, not necessarily with the final writing process, but in terms of fleshing out ideas, perhaps even giving ideas, fleshing out stories, anything in a planning process? Is there a role for AI?
Kathleen Muldoon: I don’t think this is a question of is AI good or bad. There are students who are using AI, and there are residency programs, some of which are using it to screen out applicants. And so in some ways, maybe what we will see is kind of an arms race of AI responding to AI.
But I do think that for students who feel completely lost, the first step is to just brainstorm, and maybe you need a prompt. Maybe you need some reflective process back of, “What is coming out here from this for me?” I provide that in workshops in human forms, and not just me, but a group of us. And I think that is important. But maybe there is a role for AI to say, make sense of this. Just don’t hand over the whole process.
And I am not going to shame a student who is doing that. There is a diverse number of students for whom English is not their first language, who have disabilities involved, and AI can be a great tool for a step. Just don’t polish out the parts of yourself that AI may tend to bland away.
Kevin Pho: So that actually would lead me to my next question. Like you said, where English isn’t a student’s first language, or foreign-trained students, for instance, I have seen arguments that AI levels the playing field for them. So is that a legitimate argument?
Kathleen Muldoon: So I have heard a couple of things. I have heard that some of the algorithms that are used to detect AI differentially pick up people who have used it for whom English is not their first language. So I think the evidence shows it can go both ways. It can be helpful. But these detection algorithms, or even some that are used in the screening process, tend to pick out turns of phrase as being generative AI, and maybe put them into a different pile, or differentially affect people who are coming to the process from diverse backgrounds. And so I think there is an evolving body of literature on that.
I do always caution, especially for people in the audience listening to this, people who use algorithms to screen out applicants, that those algorithms are sometimes just not very good, and they are evolving. Just be aware that there is a caveat on those processes.
Kevin Pho: So do you think that there will be a time where AI is so powerful that it could really seamlessly replicate human writing, especially when it comes to personal statements? Because the growth is so exponential. The latest models, they had to be banned by the government and regulated, and I feel that there is going to be a point relatively soon where it is going to be extremely difficult, if not impossible, to suss out AI writing.
Kathleen Muldoon: Yes, I think that is probably true. And what remains is not is AI good or bad, or should we be using it, but what are you losing when you outsource those human parts of medicine? So for example, it could get that far, to mimic human voice, and maybe your own personal human voice, and maybe you train it in that way. And in doing so, are you still outsourcing that reflective process that allows you to understand your why? I see it as kind of a protective factor. If you pretend to be somebody else and end up in a program that is not a good fit for you, or something that you never really processed or thought hard about, is this really what I want? Which comes out in that reflective process.
I see those physicians, not even that long into their careers, burning out, wondering why this doesn’t match their why.
So it could be that the process does get that far, but if you are not practicing understanding your own authentic voice, you lose that part of knowing who you are. And I would say that then you lose that part that can recognize who your patients are and be present for them, who will not have AI as a shield on their worst day, at the time when you are talking to them, in their rough language around why something is so upsetting for them, or describing their symptoms.
So again, I come back to this idea of, it probably is happening, and it probably is this arms race. But that human-to-human connection is what makes humanity and medicine, what differentiates it as an art, and not just entering your symptoms into the algorithm and getting a diagnosis back.
Kevin Pho: We’re talking to Kathleen Muldoon, certified coach and professor. Today’s KevinMD article is “Why ChatGPT can’t write your residency personal statement.” Kathleen, as always, let’s end with your take-home messages to the KevinMD audience.
Kathleen Muldoon: My take-home message for the clinicians in your audience is that I don’t think the question is did AI write this statement. It is more, who is the person in this writing, and does that person come through, and are you making it safe for them to come through as you are going through these applicants and the process? And so, as AI becomes more and more skilled at representing human voice, just remember to look for the person behind the writing.
Kevin Pho: Kathleen, thank you again for sharing your perspective and insight. Thanks for coming back on the show. I really appreciate your time.
Listen and followApple PodcastsSpotifyYouTubeMore apps · Browse all episodes
