A physician questions the COVID vaccine data from Israel

Bloodletting may be the best-known example of a once widely used, faulty medical treatment, but there are many more. Hormone replacement therapy for postmenopausal women, famously touted by experts, turned out to be complicated and sometimes harmful.

The list of medical reversals is long, and one cause is common. When observational studies compare people in the community who receive treatment with those who don’t, the treatment often looks rosy. Later, randomized trials, where people are randomly assigned to the treatment or a placebo, clean up the mess.

Observational studies are fraught, it turns out, because there’s always a reason some people receive treatment in the community when others don’t. For example, in early hormone replacement studies, women receiving the drugs were, on average, wealthier, healthier, with better access to care. These characteristics (not the drugs, as trials showed) meant the women were less likely to have heart attacks—but they were misinterpreted as showing hormone replacement therapy saves lives. However, such differences are instantly neutralized by randomly assigning people to receive a treatment or no treatment.

Therefore, it is a simple and historical fact that observational studies, no matter how rigorous, can never be counted on to find what randomized trials would find because they can’t correct for these built-in biases.

Until now.

Last week the New England Journal of Medicine published an observational report from Israel, where roughly 90 percent of citizens over age 60 were vaccinated in a national effort to interrupt a COVID surge. Incredibly, the study finds precisely—precisely—the same results as Pfizer’s randomized trial of the vaccine.

The new study finds 94 percent vaccine efficacy; the trial found 95 percent. It also finds the vaccine effect starts 12 days after the first shot—exactly, to the day, what the trial found. Before day 12, the groups have identical COVID-19 infection rates, just like in the trial. One could literally overlay most findings from the Israeli study onto the Pfizer trial.

Is this a paradigm shift? Have the researchers at Clalit, one of Israel’s four large health services, truly cracked the code—designing and executing observational studies so perfect that randomized trials may no longer be necessary?

The website OurWorldInData, which has been publicly tracking and posting Israeli COVID-19 data, offers an interesting contrast, examining the data nationally rather than just from one health system. In one instructive graph, titled “New hospitalizations for COVID-19 by age,” citizens over age 60 are divided into those vaccinated “early” versus “late.”

The graph shows that on December 18th—before Israel’s vaccination program began—the “late vaccinated” group over age 60 was already experiencing nearly three times more hospitalizations than the “early vaccinated” group. This is a jarring contrast to the Israeli study, which suggested the early and late groups are essentially identical before receiving the vaccine.

The large difference between early and late groups in national data is far more typical of observational studies. Perhaps citizens were vaccinated later because they were more hesitant about vaccination, or less healthy and mobile, or more obese, or otherwise socioeconomically at higher risk. Either way, one thing is clear: hospitalizations for COVID-19 were wildly different between groups long before vaccinations began. (It is also notable that, relative to each other, the two groups begin, traverse, and end at nearly the same place, reflecting little or no impact of vaccination).

Observational studies from England, Scotland, and even others from Israel, show many of the same differences but come to conclusions that are all over the board. Some say the vaccine works after one shot, some say not until after a second, some say 50 percent efficacy, others say 90 percent. Such is the nature of observational data: inconsistent and full of hidden differences that can mislead.

So how did the Clalit researchers achieve observational study nirvana? They would say by carefully matching each vaccinated person in their study to an unvaccinated person with similar characteristics. Of course, matching in observational studies is not new. Propensity scoring and other “adjustment” tools have long allowed researchers to even out risk differences between groups (in the hormone replacement studies results were just as wrong after adjustment).

But the Israeli researchers appear to have accomplished freakishly perfect matching. They matched almost 800,000 vaccinated people to an unvaccinated person with virtually identical characteristics. Though it’s difficult to understand how. Finding two people who match perfectly across all of their seven categories (age, sex, religion, COVID risk factors, neighborhood, prior flu shots, and pregnancy) is very difficult—but they did it 800,000 times. And they only had two unvaccinated people to choose from, on average, for each vaccinated person. But wait, there’s more: The two groups then matched almost perfectly across an additional 21 medical conditions.