Combine a hot start-up valued at 9 billion dollars, an articulate Stanford drop-out CEO with celebrity-like press coverage, and a recent Wall Street Journal report that calls into question the reliability of the company’s “revolutionary” testing, and you have the perfect storm to set the internet abuzz about Theranos.
Amidst the melee of accusations, “change-the-world” start-up jargon, and anecdotes of incongruent lab results, something has been missing: an organized, objective look at the fundamental questions and their significance. So putting aside for a moment the complex back stories of disgruntled former employees, patent authorship questions, and the “the most illustrious board in U.S. corporate history,” let’s focus on a few of the fundamental questions:
1. Has Theranos been reporting unreliable results? As compelling and concerning as claims of discrepant lab results can be, they should be considered just stories and anecdotes until we have more information. There will always be some rate of “abnormal” lab results simply due to inherent test imprecision and not due to underlying pathology. That is to say, if enough lab tests are performed, someone will always experience an incongruous result. And this is not just a new problem created by “disruptive” diagnostic companies.
Different FDA-approved instruments at different labs regularly give discrepant results, differing by as much as 25 percent or more for the liver enzyme activity tests mentioned in the Journal. Validation thus must involve multiple factors and not just comparison of Theranos to a hospital lab. These factors include the establishment and reporting of a laboratory-specific reference range, internal reproducibility, and comparison to other labs using the same instrument (a notable challenge for Theranos, who does not have an appropriate peer group for comparison).
Regarding Theranos, the most important problem is not the deviation of Theranos’ results from those of the hospital lab, but the deviation from their own results on repeat testing. Unfortunately, Theranos is yet to release any of the data that they have most certainly generated for CLIA-certification and for the voluntary submission of all their tests to the FDA. So while there are clearly some red flags, there simply isn’t yet enough information on either side for a definitive answer about the analytical reliability of the Theranos testing until we see more data.
2. Did Theranos “break the rules” in its handling and reporting of proficiency testing samples for regulatory approval? While I certainly don’t claim to know the internal details, let’s frame the discussion with a bit more about laboratory certification and “proficiency testing.” The CLIA-certification that Theranos advertises on its website indicates approval by the state and the Center for Medicare & Medicaid Services (CMS) to accept human samples for diagnostic testing. As part of this certification, CMS monitors laboratory performance through proficiency testing (PT), which involves the testing of reference samples that are typically provided by a third party. One of the rules of this testing, as also pointed out in the Journal report, is that PT samples should be handled and tested in the same manner as a patient sample.
The problem for Theranos is that PT samples are typically provided in standardized tubes, with additives similar or even identical to standard blood draw tubes. These PT samples become inherently problematic for Theranos if their patient samples are drawn directly into proprietary nanotainers. While most labs can often load the PT samples directly onto their instruments just as they would a patient sample, Theranos cannot. So there is simply no way for them to handle PT samples in the exact same manner as they would handle a proprietary tube patient samples. In a larger sense, this means the system itself isn’t set up to validate a new type of technology for the validation process is inherently based on a standardized way of doing things. Theranos might have “passed” their proficiency testing by using other instruments, but such testing might not have accurately reflected the actual patient sample workflow.
This is certainly an important issue, but the extent to which this constitutes “breaking the rules” is arguably still a tricky question. In a way, the system itself contributed. There was presumably no option for PT samples to be sent nanotainer tubes and this problem could have reasonably been anticipated prior to allowing CLIA-certification. One could even speculate that the recent unannounced FDA inspection of Theranos and the pronouncement of the nanotainers as an “unapproved medical device” were at least partially in response to a realization of this particular problem.
3. Is the innovative technology at Theranos really all that innovative? Much of the press about Theranos (and presumably much of the $400 million invested so far) has revolved around the company’s “breakthrough technology.” Yet as it stands, they are only running one test on their top secret platform. This alone raises obvious questions about the technology, but after seeing titles like, “This Woman’s Revolutionary Idea Made Her a Billionaire,” I also feel compelled to point out that the idea of microfluidics-based lab testing has been around long before Holmes applied for her first patent and dropped out of Stanford in 2003. Currently, many other people continue to work on microfluidics and “needle-free blood tests” with their own ideas and approaches. They certainly have less VC funding, but there is nothing to suggest their technology is any less promising.
If and when Theranos or others can reliably test blood using finger-sticks or novel biometric sensors, they will face yet another difficult laboratory medicine problem. The interpretation of any lab test is based on more than just a number; it is also inherently intertwined with specimen type. We simply can’t always predict how lab values will change in disease with a new type of specimen — even if both tests give the same value in a healthy person. An optimistic flip side to this problem is that new technology might enable the measurement of biomarkers that could never have shown a useful change between health and disease in venous blood.
4. Moving forward, does it even matter what instrument Theranos is using? Let’s ignore for a moment the sour taste left by the “bait and switch” of a venipuncture blood draw when you expected a finger-stick. I would argue that it doesn’t really matter how they run the test (as long as the results are reliable of course). In many ways, the real “innovation” pushed forward by Theranos is the so-called “democratization of the lab test.” The company’s mission statement doesn’t say anything about microfluidic technology; it emphasizes making “actionable information accessible to everyone.” (Of course, whether or not more lab tests means more actionable information remains controversial.) Still, with ideal implementation Theranos offers to provide easier access to lab testing, better access to results via the high-quality user experience you’d expect from a silicon valley start-up, and perhaps most importantly, transparent, affordable pricing that doesn’t change with insurance coverage and that is listed publicly right on their website.
Whether Theranos can actually deliver on these elements using standard platforms remains to be determined. Although with $400 million of investment, they can potentially afford to run at a loss for a little while as they sort out their regulatory approval — they certainly wouldn’t be the first tech startup to deliver services at a loss to gain market share and brand recognition as they develop a profitable business model. Plus, with the infamously high mark-ups for some hospital services, Theranos may be able to use traditional instruments, keep prices lower than competitors, and still stay out of the red along the way.
Dave Steiner is a pathology resident.
Image credit: Shutterstock.com
