Critics Say a Pair of California Antibody Studies Contain Critical Statistical Errors That Produced Implausible Results
Too many false positives, nonrandom study population, and infection fatality rates out of whack with other data, critics claim.
Two studies by researchers associated with Stanford University and the University of Southern California using antibody blood tests have estimated that many more people have been infected with the novel coronavirus that causes COVID-19 than confirmed diagnoses would indicate. How many more people? In the Santa Clara (Silicon Valley) study, the researchers estimated that coronavirus infections at the beginning of April were 50- to 85-fold more than the number of confirmed cases at that time. In the Los Angeles County study, they estimated the infection rate at 28 to 55 times higher than confirmed cases in that jurisdiction.
If true, these findings of vastly more widespread rates of infection would suggest that the disease is much less lethal than the crude case fatality rates suggest. (A point noted by me and other Reason colleagues in reporting on these studies.) Not surprisingly, these findings have proved quite controversial, particularly drawing the critical attention of statisticians from other institutions.
Since the Los Angeles County study has apparently not yet been published online, let's focus on the chief objections to the Santa Clara study. Those include arguments that (1) the prevalence rates among people tested for antibodies to coronavirus published in the study are mostly, or even entirely, very likely due to false positives; (2) the results are skewed because it was enriched with participants who were more likely to have been exposed to the virus than the general population of the county; and (3) that COVID-19 infections must be very widespread to produce the excess mortality seen in places like New York City, e.g, essentially most New Yorkers must already have been infected, suggesting an unprecedented level of contagiousness.
First, let's look at the problem of false positives. The researchers' blood test survey in Santa Clara County found that 1.5 percent (50 out of 3,330 people tested) were positive for the presence of antibodies to the coronavirus. So the question is, how many of the 50 positives they found might be false positives?
Critics like Columbia University statistician Andrew Gelman and Silicon Valley entrepreneur Balij Srinivasan focus on the sensitivity (true positive rate) and specificity (true negative rate) of the blood test used by the researchers in the Santa Clara study. Without going into detail, they argue that it is possible that the vast majority of the positives generated by the blood test used by the researchers could be false positives. On the other hand, the lead author of the Santa Clara study, Jay Bhattacharya, tells Science, "The argument that the test is not specific enough to detect real positives is deeply flawed."
Another problem critics allege specifically with the Santa Clara study is that the research participants were recruited via Facebook. One concern with using this recruitment method is that it might result in a group of county residents who signed up for testing because they feared that they may been exposed to the virus. Such non-random study recruitment could boost the number of positives tested, thus skewing later calculations of overall prevalence.
Finally, the researchers' estimate of an infection fatality rate (IFR) for COVID-19 of between 0.12 and 0.2 percent derived from their demographic adjustments to the raw rate of 1.5 percent positives suggests extremely high rates of infection and contagiousness. "In order to generate these thousands of excess deaths [from COVID-19} in just a few weeks with the very low infection fatality rate of 0.12–0.2% claimed in the paper, the virus would have to be wildly contagious," points out Srinivasan.
As I noted earlier, given the number of deaths in New York City from COVID-19 such a relatively low IFR would implausibly suggest that essentially every resident of the Big Apple has already been infected by the virus.
The researchers tell me via email that they are working to address these and other objections and will soon release a revised version of the Santa Clara study soon. Stay tuned.
