Sexism is a major problem in education around sciences, technology and engineering. The first step in addressing this problem is awareness.

A new study suggests many men deny that sexism is a problem in STEM fields. A study in Psychology of Women Quarterly says that that among 423 respondents in an online forum, men were more likely to suggest sexism isn’t a problem even when confronted with evidence it is.

Wow, men are really clueless. Let’s get out the cattle prods.

Or … maybe not.

A closer reading of the study reveals the way it was conducted could lead to bias. Not bias in the sense of being sexist, but biased in the statistical sense of the word, i.e. that the authors’ conclusions may not be valid.

Here’s why.

It turns out the authors collected 831 responses from an online forum regarding articles on sexism in STEM fields. We don’t know who responded, and how they might differ from those that didn’t. Were non-sexist men less likely to opine, leaving the sexist trolls to exhibit the dark side of their humanity? Probably — but we just can’t be sure because the information isn’t there.

From out of those 831 responses, the authors analyzed data only from the 423 in which the respondent’s sex could be inferred from their response. If the respondent said, “As a man, I am aggrieved that women are scientists,” then their sexist response was included. If a man wrote, “Go women in science!” without indicating his sex, his response was thrown out. Thus, importantly, the authors did not systematically evaluate the major variable of interest – respondent sex – and as a result their analyses were potentially biased to include people who mention their sex when writing about sexism.

If, for instance, male sexists are more likely to mention that they are men, then the study conclusions are completely invalid. Maybe they are, maybe they aren’t.

The authors do cop to some of the study limitations, though not this one. They write, regarding study limitations, that the study approach,

… necessitated the usage of a non-random, self- selected sample of commenters … Commenters were limited to those who had access to the Internet, were sufficiently interested in bias in STEM to read an article on the topic, read one of our three selected articles, and decided to leave a comment. As such, although our sample may be representative of the focal underlying population of individuals who are interested in such issues, as well as read and comment on such articles, these commenters may not be representative of other groups.

True enough, but how can the authors conclude they know how aware men (or rather men online) are of sexism in STEM if they do not even know how many of the original 831 respondents are men? Similarly, how can they conclude that men are more likely to be skeptical of sexism in STEM fields if they cannot compare all men’s responses to all women’s responses.

Short answer: They can’t.

What is sad is that the authors can cite a bevy of other articles that do suggest some men in STEM fields are sexist, and that some men deny sexism. I don’t doubt it: I’ve seen both.

Here’s a random sampling of anonymized Twitter commentary about the findings:

I would guess that the lived experience of these and other commentators, like mine, supports the idea that some men disregard sexism even when it’s staring them in the face.

From there, when a study comes along that supports that already-held belief, Twitter lights up. Even if the study itself is pretty poor proof.

We should continue to call out and fight sexism in STEM fields. It is pernicious, archaic, and it must go. While fighting that noble fight, which we are winning, we should not undermine the effort by basing conclusions on flawed data.

Timothy Lahey is an infectious disease physician and medical ethicist who blogs at [M U R M U R S].