A few weeks ago I wrote a piece about performance metrics in medicine. People asked me: “Well, if you don’t like the metrics, what would you use?” So I thought about this, and the best way I can think of to explain what I mean is to use an example from a different field: education.
Standardized testing has become ubiquitous in schools. The numbers generated are used by administrators to judge quality. It’s easy to use as a metric because you get a nice quantifiable number that you can compare directly with other nice quantifiable numbers. My kids have to take these tests starting in 3rd grade (about age 9). From age nine on, children are being used to generate numbers for the adults. Specifically, for people who make the rules and do the hiring and firing. What’s in it for the kids?
Well, proponents would say that it helps teachers measure students’ progress. I guarantee you the teacher every third grade classroom knows how each student is progressing, seeing as that teacher is with the students every day. A number on a test won’t surprise her (or him). Additionally, the teacher is in a much better position than a computer to understand why a student is underperforming. The same is true, or should be true, for parents. Most parents, at least in elementary school, will already be well acquainted with the educational progress of their kids. The number won’t surprise them either. It could make them feel competitive with other parents for whose kid had the highest scores, but that doesn’t benefit the kid.
Proponents say that testing allows teachers to know if they are meeting educational benchmarks. Believe me, they already know. A third grade teacher teaches third grade every year. He knows what the benchmarks are. Even if every single one of the kids in the teacher’s class are performing below the benchmarks, he knows that they are and most likely also knows why.
Some will say that testing promotes learning. Yes, certain kinds of tests, used in specific ways, can be very useful learning tools. But it has been known since at least 1988 that standardized tests don’t measure cognitive function. They measure how well a kid has memorized something, but can tell you nothing about whether or not the kid understands what he learned or if he can apply the new knowledge to his existing knowledge to solve problems. Also, the time required for administering these exams takes time away from the activities that would have helped that kid understand what he has memorized.
So getting numbers from kids doesn’t really benefit them at all. It doesn’t help teachers or parents much. It is great for administrators. If I enroll my kid in a new school, am I going to ask the principal how the school did on standardized testing? Maybe, but that only gives me an idea if the students in the school do well on standardized tests. That’s all it means. In order to find the best teacher for my kid, I’m going to ask other parents. “Hey, my kid is going into first grade this fall. Who do you recommend for a teacher? Who’s the best? My kid has X personality, just like your kid. What teacher was the best fit for your kid when she was in first grade?” Other parents will know, I guarantee. They are in the best position to know. They have seen the teachers work. In my public school, and I guess in most, parents don’t get to choose the teacher. But what if they were? It would be immediately clear to everyone who the good teachers are. No numbers required.
Let’s look again at the 30-day mortality metric. That number is acquired from hospital data, and as we have seen, can be gamed to some extent. We also know that the 30-day mortality measures only how many patients died within 30 days. That’s it. There are no qualifiers in there for degree of illness, severity of atherosclerosis, co-morbid conditions, etc. So you might know the number, but it doesn’t really tell you anything. It doesn’t help the surgeon, who knows what is number is compared to other doctors and knows what the benchmarks are. He wouldn’t use his number to measure how good a surgeon he is. He’d use it to compete with other surgeons. The number doesn’t help the patient in an emergency situation since the patient doesn’t get to decide who his doctor is. The only people the 30-day metric helps is administrators and payors.
How do you know who the good surgeons are? Ask their colleagues. Not their fellow surgeons. Ask the nurses they work with, the anesthesiologists or critical care doctors or radiologists. And don’t just ask them who is the best, but ask them, “Who would you want to do your own surgery?” Believe me, they know. They are in the best position to know. I personally know who I would want to take out my appendix or gallbladder, should I need it, and would wait a good long time to get the surgeon I want. Do I know this doctor’s 30-day metric? Not at all. I’ve seen him work.
Shirie Leng, a former nurse, is an anesthesiologist who blogs at medicine for real.
Image credit: Shutterstock.com
