We all need hope this holiday season, but as an experienced physician, biostatistician, and research methodologist, it is my duty to share why the encouraging vaccine results reported by pharma companies in the last month must be applauded but interpreted with caution.

Pfizer/BioNTech and Moderna are both to be commended for successfully recruiting tens of thousands of subjects in a remarkably narrow window of time. As a clinical trialist who has helped set up large international randomized trials, I can vouch that this speed and scale are unprecedented.  Despite the divisiveness of our political landscape, Americans can certainly be proud that their tax dollars under Operation Warp Speed and their willingness to be volunteers for these trials at a moment’s notice made this record-setting pace possible.

And while these early positive data are exciting, there are a few caveats: First, it’s far too early to completely embrace or condemn non-peer-reviewed Industry reports. Vaccine studies are designed for 2-years of data, not two months.  Adequate follow-up is essential, as is the genuine need for 6-9 months of exposure data for enough subjects to acquire either an infection or an adverse reaction. Despite the impressive speed of trial initiation, we must be patient.  Experience from decades past tells us that vaccine trials require time to properly demonstrate safety and efficacy and then additional time for unbiased, objective, external peer-review.

Second, the two preliminary mRNA vaccine reports quote efficacy numbers in the 94% range (as if they are both accurate and conclusive), essentially ignoring any information from the other 99.8% of those enrolled.  For example, in the Moderna trial of 30,000 subjects, only 95 (0.3%) have a symptomatic infection so far; hence, 99.7% are asymptomatic, but not necessarily SARS/CoV-2 free.  Given interim placebo and vaccine success rates of 99.4% and 99.97%, respectively, the absolute reduction in risk for the overall study population at 2 months is less than 1%.

Whenever we hear results from large studies, we expect the numbers of ill subjects to be much larger than any percentages quoted.  Be skeptical of tiny “p values” and percentages bereft of their limitations. Estimates should include error terms that describe a range wherein the true effect lies. For example, 95% confidence intervals (95%CI) describe the range of possibilities, allowing us to be 95% certain the true value is contained somewhere within that range. The 95%CI for each study arms’ infection rate should not overlap when they are significantly different. Quoting precise point estimates to 2 decimal places without an error term presumes a degree of precision currently lacking.

I’ve no doubt all of these companies have a sincere desire to see their vaccines save lives. But despite my belief that they will eventually show an impact on mortality, that hasn’t happened yet. Quoting oversimplified results via national media undermines the promise these hard-working corporate and academic teams made only a few months ago not to cut corners and not to bow to political pressures to release the results early (hence, the clever announcement timing by Pfizer just after a national election).

Exercising restraint and revisiting that promise during the upcoming FDA review would be prudent.  As a medical scientist, I am willing to be part of the trials and willing to personally receive an experimental vaccine, such as they all are at this point. But I must object to governments doing an end-run around science. Americans should not be distracted by the premature vaccine approval in the UK, which was enacted under considerable duress in the wake of poor governance and public health policy failure.

This is not to suggest in any way that the need for a SARS/CoV-2 vaccine is overstated, as was perhaps the case of the 1976 Swine Flu vaccine during the Ford administration. But credibility is on the line in a Nation that does not trust the government.  One need only recall the manufacturing mishaps with the Salk and Sabin polio vaccines to appreciate the need to put safety first.  We must run similar quality control checks at the manufacturing facilities that are proposing to roll out these vaccines by the billions. I have worked with many companies in Pharma where the Phase1-3 studies worked great, but the facilities in charge of scaling production had issues that ultimately halted full approval. Thankfully.

I want these vaccines for all of us; as someone who worked in the hot zone of a New York ER in April, I know firsthand of COVID death and devastation.  As someone who contracted the virus through my job, I know it is not the bloody “flu.”  We need desperately need vaccines. But experts other than me have already suggested we should wait to see significant and sustained effectiveness at 6-12 months with no sign of significant adverse events before our FDA proceeds with approval.  Premature approval (EUA or otherwise) would make it appear unethical to randomize patients to placebo and bring all competing trials to a screeching halt.

A safe solution

A better short-term option, by far, would be a compassionate use license that allows early access to those at especially high risk.  However, data should be kept on these “compassionate use” patients to more carefully determine real-world effectiveness (Phase IV).

This would allow time for: current trials to continue, manufacturing facilities to be inspected, and both the null hypothesis and global public health respected.

We are genuinely thankful for the thousands of globally-minded citizen volunteers for these studies. We are even more grateful that so few of these brave trial volunteers became ill.  While these novel mRNA vaccines certainly appear to work in the short term, we have, to date, only a limited idea of how well and how safe.

We do know that we’re seeing more cases currently; our neighbors to the North saw a COVID-19 spike in the wake of Canadian Thanksgiving. The post-Thanksgiving USA sees similar spikes in many places.  As a trained biostatistician, chemical engineer, and medical researcher, I remind you of Aesop’s tortoise and hare. As a frontline emergency specialist (ER doc), I beg you to please mask-up and adhere to these “irritating” public health measures this holiday season and for the foreseeable future.  Vaccinated or not, we want you to appreciate the importance of our relationship with each other in health and illness.

G. Luke Larkin is a physician-scientist.

Image credit: Shutterstock.com