False Negatives, False Positives, and Scholarly Journals

It is law review submission season, so I have had many conversations lately both on and off of social media about the student-run law review system. This recent poll by Professor Derek Muller was interesting, for example.

The quality of law review articles in top law journals is ___ it was ten years ago.

— Derek T. Muller (@derektmuller) February 22, 2024

I think it is a given that everybody thinks scholarly journals should publish better scholarship, and not publish worse scholarship. But as I thought about Professor Muller's question it seems to me that it's important to distinguish between the two different kinds of mistakes a journal can make: false positives (i.e., publishing something bad) and false negatives (i.e. declining to publish something good).

Given that no system is perfect, there is obviously some tension between these two goals. If you want to emphasize avoiding false positives then you should probably want a field in which there are a small number of highly-regarded journals with rigorous peer review on the basis of rigorous methods. Many potentially "good" pieces may get screened out of this system, but if something is published in the field's top journal you can basically take it to the bank.

On the other hand, if you want to emphasize avoiding false negatives then you should probably want a system closer to the current law review system, with many journals applying a much more pluralistic conception of merit, chasing pieces quickly through simultaneous submission. Most pieces that are significant and relevant can find a decent home.

Now neither system is perfect even at the goal it is trying to maximize -- there are rumors of corruption in even the most rigorously peer-reviewed fields, and there are still excellent pieces of legal scholarship that somehow don't fit the fashions of student editors. (And for that latter case may I again recommend the Journal of Legal Analysis, a peer-reviewed law review at Harvard Law School where I serve as a co-editor, especially for public law pieces?).

But I think it's helpful to articulate this distinction and these tradeoffs, and to remember that some changes that would fix one of these problems would make the other much worse. (For instance, returning to Muller's poll question, my hypothesis is that the law review system, taken as a whole, has perhaps gotten slightly better at avoiding false negatives (i.e. finding a home for good pieces) even if it has perhaps gotten slightly worse at avoiding false positives (i.e. letting bad pieces into good fora).

[For previous posts on law reviews, see here and here.]

UPDATE: An important and interesting corollary is the role of so-called specialty journals —the University of Chicago Business Law Review, the Harvard Journal of Law and Public Policy, the Yale Journal of Law and the Humanities, the Yale Journal of Law and Feminism, and so on. It seems to me that such journals are important for any areas where the flagship law reviews do have an unusually high number of false negatives.

For instance, one hears claims that the flagship law reviews do not accept (perhaps because they cannot adequately appreciate) even excellent pieces about tax law, private law, etc. Similarly, journals that focus on conservative thought, feminist thought, etc. may reflect a view that the flagship journals are insufficiently appreciative of good work with those attributes. These specialty journals provide another useful backstop against the false-negative problem in law review publishing.

At the same time, there might be specialty journals in areas that once had a great false-negative problem but where flagship journals as a whole have since started to self-correct. The Duke Journal of Gender Law and Policy, which ceased publishing in 2020 after 25 years may be an instructive example. In the Foreword to the final issue, Dean Kerry Abrams observed:

Unlike other journals that we have launched and then decommissioned, however, the gender journal has an additional, equally positive thread in its story . . . Gender, once a subject ignored or outright avoided by mainstream law journals, has now become a much more common subject for law reviews. In fact, in the last ten years, flagship law reviews have published the most influential and highly cited articles on gender and antidiscrimination law,[7] criminal law,[8] family law,[9] religion,[10] citizenship,[11] and abortion.[12]

7. See, e.g., Cary Franklin, The Anti-Stereotyping Principle in Constitutional Sex Discrimination Law, 85 N.Y.U. L. REV. 83 (2010).

8. Melissa Murray, Marriage as Punishment, 112 COLUM. L. REV. 1 (2012).

9. Clare Huntington, Postmarital Family Law: A Legal Structure for Nonmarital Families, 67 STAN. L. REV. 167 (2015).

10. Douglas Nejaime & Reva B. Siegel, Conscience Wars: Complicity-Based Conscience Claims in Religion and Politics, 124 YALE L.J. 2516 (2015).

11. See, e.g., Kristin A. Collins, Illegitimate Borders: Jus Sanguinis Citizenship and the Legal Construction of the Family, Race, and Nation, 123 YALE L.J. 2134 (2014).

12. Linda Greenhouse & Reva B. Siegel, Before (and After) Roe v. Wade: New Questions About Backlash, 120 YALE L.J. 2028 (2011).

If specialty journals are sufficiently successful in generating and demonstrating the importance of excellent scholarship in their domain, it is possible they will become less necessary to the next generation of scholars.

Overall, specialty journals have an important role to play especially in areas of scholarship with systematic false negatives.

Comments

[Rev. Arthur L. Kirkland]

Is that the Journal of Legal Analysis that was an Olin-class part of the separatist clingerverse?

[Rossami]

Interesting perspective. I hadn't really considered this in the context of law review journals but this is a crushing problem in the context of medical and scientific journals. And in those contexts, readers commonly assume that academia has settled on the low-false-positive model yet the reproducibility crisis suggests that we've actually settled on the low-false-negative model. This deserves more thought...

[Joe_dallas]

The issue of false positives is much more prevalent with the agenda driven studies / political driven studies

High success rates with transgender transitions
12% asthma cases associated with gas stoves
high effectiveness of masking
to name a few,

[Life of Brian]

A few years ago these chaps had a good run getting deliberate satirical nonsense on nouveau gender and sexuality issues accepted and published by a number of peer-reviewed journals. The (predictably defensive) responses from a number of the journals essentially blame the authors for telling them what they clearly wanted to hear:

Ann Garry, the interim editor of Hypatia: A Journal of Feminist Philosophy, told AFP that she was "deeply disappointed" to learn about the hoaxes her journal published.

"The idea that individuals would submit fraudulent academic material violates many ethical and academic norms," she said.

Roberto Refinetti, editor-in-chief of the journal Sexuality and Culture, told AFP that the article on dildos "was reviewed by three university-affiliated experts in the field, none of whom suspect a hoax."

Refinetti was equally defensive. The fabrication by the authors "speaks against their integrity, not against the integrity of the journal that published the findings," he said.

[Joe_dallas]

I will let Don N correct and clarify my comment, though regarding my comment on the positive outcomes of transgender transition studies. Those study suffer from numerous problems such as very low response rates, most of which have response rates below 50% in a subpopulation that requires lifetime mental health care and lifetime medical care due to transition surgeries. - how do you lose contact with 1/2 of that small subpopulation. Surveys conducted by groups with high bias and high levels of conflict of interest. (ie vested interests).

Perhaps its that those studies dont get published (or submitted) to high quality journals. The vast majority of my negative comments on studies deal the study results that are problematic and not very plausible. I have no issue with studies that are (or maybe) wrong but are reasonably plausible .

[David Nieporent]

So, so many areas where Joe_dallas is an expert.

[Don Nico]

I don't see any reason to be insulting about his comment.

Regarding masking, for example: While careful physical measurements of transmission probability versus particulate or aerosol size can be accurate, simple observation of masking practices in an airport will more than suggest. That physics experiments don't translate well to control of infections, especially when a virus is very highly contagious.

The discrepancies between journal studies and results in practice are not the fault of the reviewers. In the medical papers that I am asked to review, I have found that the researchers have been careful to describe the limitations in their data sets and to caveat conclusions outside the clinical setting. Generally the journal can take some credit for encouraging such careful professional practice.

[Don Nico]

“If you want to emphasize avoiding false positives then you should probably want a field in which there are a small number of highly-regarded journals with rigorous peer review on the basis of rigorous methods. ” That is at best approximately valid. Even the best journals in the hard sciences, one finds numerous manuscripts of which one ca say at best that they are plausible and certainly not obviously false. Whether those papers stand the test of time over a decade in the most hotly investigated areas is less probable although the chances are likely to be over 60%.

The question of false positive versus false negatives is especially difficult with respect to theoretical analyses. Consequently for the journal that I edited for decades, we strongly prefer to see analyses supported by adequate experimental validatation.

Rigorous peer review works okay but far from perfectly, it is also in tension with the expectations of authors to has a decision on the manuscript within a month.

[Joe_dallas]

good comment