Lemoine on the Use and Abuse of Statistics

Philippe Lemoine has a new post called “Lockdowns, econometrics and the art of putting lipstick on a pig.” He takes apart the paper “Causal impact of masks, policies, behavior on early covid-19 pandemic in the U.S.” by Victor Chernozhukov, Hiroyuki Kasaharab, and Paul Schrimpfb.

Some highlights

-The conclusions of Chenozhukov et al. rest on several dubious assumptions and are extremely sensitive to reasonable changes in the specifications of their model.

-The authors’ most robust finding is that mandating masks for public facing employees reduces the spread of COVID-19. This result is highly doubtful, given that they collected data on broader mask mandates that they chose to ignore in the paper. When the same models are run with the broad mask mandate, there is no effect.

-This is particularly troubling given that in the paper they speculate that mandating face masks for everyone might have a much larger effect, a proposition they had the data to test.

-A much simpler version of their model fitted on simulated data finds that, even under very favorable conditions, the model performs extremely poorly. This is a problem not just for Chernozhukov et al., but for any study that relies on similar models to analyze the effects of non-pharmaceutical interventions.

Philippe’s report is not simply about one paper, but highlights the flaws of research based on statistical models that rely heavily on the questionable assumptions of authors. The incentive structure of academia rewards overly broad claims based on flimsy evidence, particularly when those claims are justified by complex models most people do not understand. A broader skepticism of supposed findings based on such models that can’t be verified through more straightforward methods is therefore warranted.