The Bad Science Behind Jonathan Haidt's Call to Regulate Social Media

In his 1996 book, The Vision of the Anointed, economist Thomas Sowell sketched out a pattern that many of the "crusading movements" of the 20th century have followed. First, they identify a "great danger" to society, followed by an "urgent need" for government action "to avert impending catastrophe."

A new book by psychologist and author Jonathan Haidt, The Anxious Generation, argues that the government must regulate social media because it's causing a teen mental health crisis. Haidt is, in many ways, a model researcher because of his rigor, transparency, and openness to dissent. On this issue, however, he fits neatly into Sowell's framework.

Those best equipped to get attention from the government and the media are the most "articulate" people, Sowell observes, and they often reference opaque studies without explaining them. And Haidt is certainly articulate—his book is well-written and filled with compelling insights. But he claims far too much certainty for his views, based on research that is mostly junk. And he advocates for restrictive government policies without doing the simple tests that might support or disprove their value.

Academic studies often make use of statistical techniques that are hard for the average person to decipher, which is a shame because "most published research findings are false," as Stanford's John Ioannidis argued in a 2005 paper. Ioannidis wasn't just referencing the many scandals of fabricated data, conscious or unconscious bias, and misrepresented findings. Even top researchers at elite institutions have been guilty of statistical malpractice. Peer review is worse than useless, better at enforcing conventional wisdom and discouraging skepticism than weeding out substandard or fraudulent work. Academics face strong pressure to publish flawed research. Few have the skill and drive to produce high-quality publications at the rate required by university hiring and tenure review committees. Even the best researchers resort to doing some easy, low-quality studies. Bad studies tend to be the most newsworthy and the most policy-relevant.

Many of the papers Haidt compiled contained coding errors, inappropriate statistics, and other issues. Most downloaded some data of little relevance—either cheap to generate, like surveying your sophomore psychology students, or data collected for a different purpose—and analyzed it with an off-the-shelf statistical approach.

Haidt cites 476 studies in his book that seem to represent an overwhelming case. But two-thirds of them were published before 2010, or before the period that Haidt focuses on in the book. Only 22 of them have data on either heavy social media use or serious mental issues among adolescents, and none have data on both.

There are a few good studies cited in the book. For example, one co-authored by psychologist Jean Twenge uses a large and carefully selected sample with a high response rate. It employs exploratory data analysis rather than cookbook statistical routines.

Unfortunately for Haidt, that study undercuts his claim. The authors did find that heavy television watchers, video game players, and computer and phone users were less happy. But the similar graphs for these four ways of spending time suggest that the specific activity didn't matter. This study actually suggests that spending an excessive amount of time in front of any one type of screen is unhealthy—not that there's anything uniquely dangerous about social media.

An example of a bad study that Haidt cites in his book is one that paid $15 each to 1,787 self-selected internet respondents, aged 19 to 32, to answer 15 minutes' worth of questions online. Few were likely to have been teenage girls, and there's no reason to expect any were depressed teenage girls who used social media. In fact, I couldn't find any studies in Haidt's compendium that spent substantive time interviewing any depressed teenage girls or heavy social media users.

Another study that uses low-quality data was based on surveys of 143 University of Pennsylvania students, who participated for psychology course credit. Undergraduate psychology students at an elite university are hardly a representative sample of the population. The authors seem to have made significant coding and random assignment errors.

Haidt cites 17 studies he considers to be longitudinal that either find no effect or an effect in the opposite direction of his claim, and only four were true longitudinal studies, meaning they analyzed the same group of people at different times to see how changes at one time, like increased social media use, were associated with future changes, like more depression. One of the studies on Haidt's list contradicted his claim, finding that depression occurs before social media use, not the other way around.

Practically all of the studies Haidt cited either have major methodological errors or didn't say what he claimed. I doubt many of the senators on the Judiciary Committee or members of their staff, to whom Haidt claimed an "urgent need" for government action, read the underlying papers. The fact that Haidt cites a lot of studies just makes the problem worse. Errors cascade; they don't cancel each other out.

Even the best studies Haidt relies upon have the fatal flaw of not studying the subject. You can't establish the effect of heavy social media use on teenage girl depression unless you study heavy social media users and depressed teenage girls. None of the studies do this. Instead they study mostly adults, mostly average social media users, without serious psychological issues.

When Haidt moves on to policy, a different kind of study is called for. If you want to ban phones in schools, study kids who went to phone-free schools vs. a control group of kids who were allowed to use phones in school. Even this would only tell you average effects, but at least showing the change is positive on average is a step toward testing your hypothesis.

To be fair, Haidt does have some solid policy proposals that don't suffer from this flaw. When he argues that governments should stop criminalizing free play or that internet companies should insist on more reliable age-verification measures, which parents are requesting, he doesn't need studies.

When Haidt first presented his argument on Substack, I critiqued the studies he cited, and he responded by suggesting that I have impossibly high standards for proof and that the consensus of hundreds of researchers studying related issues is strong enough to justify policy actions. My objection is that the researchers whose work Haidt relies on didn't bother to talk to teenagers who are heavy social media users and who are in treatment for depression. Based on poor quality studies, he wants the government to dictate to parents how to raise their children, overruling the judgments of those who are intimately acquainted with the individuals involved.

Some have argued that it's obvious that social media is causing depression in teenage girls. It may be a contributing factor, but the purpose of social science research is not to confirm but to challenge our knee-jerk assumptions, because reality is so complicated. The proper scientific approach is to try to falsify your hypothesis.

If you are going to recommend new laws to regulate social media, as Haidt has, you need to consider the likelihood that your intervention isn't going to yield the effect you want.

There's a tendency among the anointed, Sowell observes, to see "little standing between intention and result." But the real world is "a system of innumerable and reciprocal interactions, all constrained within the confines of natural and human limitations." Often "individual problems cannot be solved one by one without adding to other problems elsewhere."

As any parent knows, children are complex human beings. Policies designed to control their behaviors don't often yield the results we expect. Even if we knew social media use caused depression, we wouldn't know the effect of policies that restrict social media use.

We know alcohol leads to many catastrophic problems, but that doesn't tell us how to regulate youth drinking. If we stop adolescents from using social media, we don't know what they'll do instead, and it might be more harmful.

There's no doubt that social media has significant effects, positive and negative, on the mental health of many young people. Dealing with this is one of the challenges modern parents must face.

But Sowell's anointed class of academics seem happy to impose coercive solutions on evidence that takes a one-size-fits-all approach and that they wouldn't rely upon for their own troubled teenage daughters.

Photos: Daniel Hambury/Eyevine, Mikkel Aaland/Free To Choose, E. Jason Wambsgans/TNS/Newscom, CNP/AdMedia/SIPA/Newscom, BONNIE CASH/UPI/Newscom, Richard B. Levine/Newscom, Howard Lipin/TNS/Newscom

Music: "Stream" by ANBR, "Empty Rooms" by Gal Lev, "Eclipse" by Yuriy Leontiev, "Curiosity" by Kevin Graham, "Discovery" by We Dream of Eden, and "Continent' by ANBR. Roadway by Ardie Son

• Video Editor: Adani Samat
• Audio Production: Ian Keyser
• Graphics: Isaac Reese