Corpus Linguistics in Court?

Two Sixth Circuit judges debate the issue, in an opinion filed today.


See the concurring opinions of Judges Jane B. Stranch and Amul Thapar, in Wilson v. Safelite Group, Inc. (For an earlier example of such a debate in a judicial decision, see here; for guest posts by Justice Thomas Lee and Stephen Mouritsen explaining and defending the use of corpus linguistics in law, see here.)

From Judge Thapar:

I agree with the majority's textual analysis and concur to the extent the majority rests on that analysis. Because the text of 29 U.S.C. § 1002(2)(A)(ii) is clear, we should go no further. And the text is clear, as many tried-and-true tools of interpretation confirm. But so does one more: corpus linguistics. Courts should consider adding this tool to their belts….

[Corpus linguistics] draws on the common knowledge of the lay person by showing us the ordinary uses of words in our common language. How does it work? Corpus linguistics allows lawyers to use a searchable database to find specific examples of how a word was used at any given time. State v. Rasabout, 356 P.3d 1258, 1275–76, 1289 (Utah 2015) (Lee, A.C.J., concurring in part and concurring in the judgment). These databases, available mostly online, contain millions of examples of everyday word usage (taken from spoken words, works of fiction, magazines, newspapers, and academic works). See, e.g., Corpus of Contemporary American English, BYU, (listing types of sources); Corpus of Historical American English, BYU, Lawyers can search these databases for the ordinary meaning of statutory language like "results in." The corresponding search results will yield a broader and more empirically-based understanding of the ordinary meaning of a word or phrase by giving us different situations in which the word or phrase was used across a wide variety of common usages.  See Lee & Mouritsen, supra, at  831 ("Linguistic corpora can perform a variety of tasks that cannot be performed by human linguistic intuition alone."). In short, corpus linguistics is a powerful tool for discerning how the public would have understood a statute's text at the time it was enacted.

Of course, corpus linguistics is one tool—new to lawyers and continuing to develop—but not the whole toolbox. Its foremost value may come in those difficult cases where statutes split and dictionaries diverge. In those cases, corpus linguistics can serve as a cross-check on established methods of interpretation (and vice versa). See Lawrence B. Solum, Triangulating Public Meaning: Corpus Linguistics, Immersion, and the Constitutional Record, 2017 BYU L. Rev. 1621, 1669–70 (2017) ("[C]orpus linguistics allows for rigorous intersubjective validation of individual subjective judgments about word meaning."); see also Clark D. Cunningham et al., Plain Meaning and Hard Cases, 103 Yale L.J. 1561, 1566 (1994) (book review) (arguing that empirical methods may assist judges in hard cases of ordinary meaning). This cross-check can provide both judges and parties with greater certainty about the meaning of words in a statute….

Judge Stranch's response:

I write separately to express some concerns about the concurrence's endorsement of "corpus linguistics," a proposed method of statutory interpretation described in a handful of recent state court opinions [mostly from Utah]. This tool invites judges to perform the same kind of subjective decision making that the concurrence otherwise cautions us to avoid. There are  several reasons why we should decline this invitation.

The first is a practical problem. A keyword search using a corpus linguistics database  will likely result in dozens, if not hundreds or thousands, of examples of a term's usage. How should courts make sense of all this information? First, we could count the number of times a term is used in the database (assuming appropriately selected parameters) and then decide that the most frequently used meaning is the ordinary meaning. But that approach would risk privileging the most newsworthy connotations of a term over its ordinary meaning. See, e.g., Carissa Byrne Hessick, Corpus Linguistics and the Criminal Law, 2017 B.Y.U. L. Rev. 1503, 1509 ("[A] corpus analysis may demonstrate that seventy percent of all mentions of the word 'flood' occur in the context of [] superstorms. But that does not tell us whether the average American would understand the statutory term 'flood' to include three inches of water in a homeowner's basement after a neighboring water main burst."). It would also fail to cull irrelevant results. If we use the database to determine the meaning of "results in" during the 1960s and 1970s (the era closest to when ERISA was drafted), we find examples of this term's usage in contexts that bear no relationship to our own. Does it matter, for purposes of deciding whether ERISA applies to Wilson's deferred compensation plan, how "results in" was used in a book about farm animal management in 1976, or in an article from Sports Illustrated about New York's cool spring weather in 1964? I think it does not. And even if consulting this overinclusive data set might help judges "to avoid basing conclusions on a few speakers' idiosyncrasies," it is "the 'idiosyncrasies' of [Congress that] constitute the rule of law in this [country]. And the only way to identify those idiosyncrasies is through the text of the [U.S.] Code, which is wholly absent from [the corpus linguistics] data set." State v. Rasabout, 356 P.3d 1258, 1266 (Utah 2015) (call numbers and internal quotation marks omitted)). This suggests to me a disconnect between corpus linguistics and the judicial work of statutory interpretation.

Another approach would require the court to perform this culling process itself. For example, we could assume that the drafters employed popular, as opposed to technical or legal, language and decide which uses of "results in" during the 1960s and 1970s should be included in our analysis and which should not. But by what metric would we make that choice? Perhaps most could agree that a book about farm animal management is not relevant here. But what  about an article reporting a union strike? Or one about federal tax penalties?3 Such choices  would require highly subjective, case-by-case determinations about the import and relevance of a given source. Textualists have long advised us to forgo that interpretive method. See, e.g., Conroy v. Aniskoff, 507 U.S. 511, 519 (1993) (Scalia, J., concurring in the judgment) ("The legislative history of [this] Act contains a variety of diverse personages, a selected few of whom—its 'friends'—the Court has introduced to us in support of its result. But there are many other faces in the crowd, most of which, I think, are set against today's result."). Legislative history tells us, at a minimum, how some of the statute's authors understood a term; corpus linguistics does not offer even that insight.

In part because of these practical problems, the use of corpus linguistics is a difficult and complex exercise. That is why, for centuries, we have left this task to the trained lexicographers who author the tool we already employ—a dictionary. [Judge Thapar's] concurrence describes much of what lexicographers do every day.

Lexicographers engage in "research, but also decisionmaking: the primary job of the lexicographer in creating a dictionary is to determine meanings of words, and to determine what different meanings a word might have." Rubin, supra at 181. And because "[t]he line between one meaning and another is seldom clear," this process "leaves much of the final determination to the experienced judgment of the editorial staff." Id. The other concurrence argues that, notwithstanding their training, these experts sometimes select outdated or otherwise unreliable meanings for disputed terms. But I would not substitute the ad hoc selection process of individual judges for the "experienced judgment" of these trained scholars. Doing so would convert judges into armchair lexicographers, attempting the same work that dictionary authors have been performing for centuries. But unlike those experts, judges would shoulder this task without the specialized training necessary to make a reliable and neutral judgment call. Encouraging litigants to take on that same role would make the problem worse, not better.

I do not suggest that corpus linguistics can never assist judges in the difficult project of statutory interpretation. But, in the unusual case where such a tool might prove useful, I would leave this task to qualified experts, not to untrained judges and lawyers. See, e.g., Brief for Professor Clark D. Cunningham, et al. as Amicus Curiae on Behalf of Neither Party, In Re: Donald J. Trump, President of the United States of America, No. 18-2486 (4th Cir. Jan. 29, 2019) (discussing use of corpus linguistics by professor of applied linguistics to help determine the meaning of "emoluments" during the founding era). And before we add corpus linguistics to our judicial toolkit, we should first remind ourselves what our toolkit is for. I agree with the concurrence that statutes ought to give "ordinary people fair warning about what the law demands of them." United States v. Davis, 139 S. Ct. 2319, 2323 (2019). But that is the responsibility of legislators, not judges. Once the torch passes from Congress to the courts, our duty as judges is simply to determine what our elected members of Congress meant when they passed the statute—even  if that is  not  the meaning we or the public might  routinely employ.  In most cases, adding corpus linguistics to our judicial toolkit would make it harder to focus on that narrow duty. This case underscores why.  Our task here is to decide what Congress meant  by "results in" and "extending to" when it defined the universe of employee pension benefit plans covered by ERISA. The other concurrence proposes that we divine that meaning not by considering ERISA's statutory framework or legislative history, but by looking to the language of an article from Sports Illustrated and a book about farm animal management. I struggle to see why those sources would tell us as much as, for example, what the legislature told us about the structure and purpose of ERISA when it drafted the statute.

Underlying these practical usage issues is my concern with the implicit suggestion that corpus linguistics is a simple, objective tool capable of providing answers to the puzzle of statutory interpretation. The use of corpus linguistics brings us no closer to an objective method of statutory interpretation. Instead, it encourages judges to stray even further from our historic and common-sense considerations—including the "text, structure, history, and purpose" of a statute, Maracich v. Spears, 570 U.S. 48, 76 (2013) (citation omitted)—that ought to guide our analysis.

Judge Thapar's reply:

The other concurring opinion argues that we should not add corpus linguistics to the judicial toolkit for several reasons. The first is methodological—corpora are not representative because of their sources. For instance, a corpus search for "flood" may lead to an overinclusion of newspaper articles talking about giant flood waters rather than basements flooding. But the entire practice of law—and certainly the practice of interpretation—involves judgment calls about whether a particular source is relevant. And, at least with corpus linguistics, those calls  can be vetted by the public in a more transparent way. Cf. Muscarello v. United States, 524 U.S. 125, 142–43 (1998) (Ginsburg, J., dissenting) (criticizing the majority opinion for selective and non-transparent examples of word use). That is more than can be said of the alternative, which, as Justice Lee has thoughtfully noted, is for a judge to use his or her intuition—something far less representative and frankly far less "democratic." See Rasabout, 356 P.3d at 1274–75 (Lee, A.C.J., concurring in part and concurring in the judgment). Plus, the danger of judges relying upon their own intuition is that we introduce other risks, like confirmation bias. Id. at 1274. Judges may unintentionally give greater weight to those definitions that match up with their preconceived notions of a word's meaning. We cannot get away from confirmation bias altogether, but we can surely check our intuition against additional sources of a word's meaning. The corpus allows us to do this.

Second, the other concurring opinion argues that the use of corpus linguistics will descend into mere rote frequency analysis; judges will simply pick the use of the word that shows up the most. Yet judges who use corpora do not become automatons of algorithms. They will still need to exercise judgment consistent with the use of the other tools of statutory interpretation. Sometimes the most frequent use of a word will line up with its ordinary meaning as used in a statute. Sometimes it will not. The data from the corpus will provide a helpful set of information in making that interpretive decision. But the judge must make the ultimate decision after considering multiple tools.

Third, the other concurring opinion suggests that corpus linguistics is redundant when compared with another tool—dictionaries. Expert lexicographers already do corpus linguistics when compiling dictionaries, so, the argument goes, when judges use corpus linguistics, they become unnecessary and unhelpful armchair lexicographers. But the use of corpus linguistics improves upon dictionaries by helping pinpoint the ordinary uses of a word at the time a statute was enacted. For example, when a court considers a dictionary definition, it looks at a dictionary from that time period. See New Prime, 139 S. Ct. at 539–40 & n.1. But the usage examples in those dictionaries  often  come  from  a  time  before  the  dictionary  was  published.  See  Lee  & Mouritsen, supra, at 808–09; Henry M. Hart, Jr. & Albert M. Sacks, The Legal Process: Basic Problems in the Making and Application of Law 1190 (William N. Eskridge, Jr. & Phillip P. Frickey eds., 1994) ("An unabridged dictionary is simply a[] historical record, not necessarily all-inclusive, of the meanings which words in fact have borne, in the judgment of the editors, in the writings of reputable authors."); see also id. at 1375–76. So the dictionary definition may actually tell us the ordinary meaning at a time long before Congress enacted the statute. See Lee & Mouritsen, supra, at 809; Scalia & Garner, supra, Appendix A at 419 (noting that dictionaries lag behind the times). And in many cases (like the ones discussed below), both the majority and dissent will point to dictionaries without any clear resolution. Instead of relying on just a few sample sentences in the dictionary, the corpus develops a broader picture of how words were actually used when Congress passed the statute.

Plus, "[w]e judges are experts on one thing—interpreting the law." Rasabout, 356 P.3d  at 1285 (Lee, A.C.J., concurring in part and concurring in the judgment) (emphasis omitted). Corpus linguistics is just one variation on a very old theme in this field of expertise. Judges  often consider the context of words—both within and beyond the text of the statute in dispute. See Scalia & Garner, supra, § 31 (detailing the "noscitur a sociis" canon of interpretation); see also Caleb Nelson, What is Textualism?, 91 Va. L. Rev. 347, 355 (2005). Judges look to contemporaneous judicial decisions. See, e.g., New Prime, 139 S. Ct. at 540. They look to seemingly common phrases. See, e.g., FCC v. AT&T, Inc., 562 U.S. 397, 403–04 (2011) (considering how the word "personal" is used in "personal life" and "personal expenses"). And, for constitutional cases, they look to word use in the Anti-Federalist and Federalist Papers. See, e.g., United States v. Lopez, 514 U.S. 549, 586 (1995) (Thomas, J., concurring). While sometimes this "enterprise may implicate disciplines or fields of study on which we lack expertise, [it] is no reason to raise the white flag" and forgo considering corpus linguistics. Rasabout, 356 P.3d at 1285 (Lee, A.C.J., concurring in part and concurring in the judgment).

Instead, judges should do what they have always done—"summon all our faculties as best we can" and take advantage of adversarial briefing. See id.

In sum, I agree that corpus linguistics is not the only tool we should use, but it is an important tool that can assist us in figuring out the meaning of a term.

Thanks to Howard Bashman (How Appealing) for the pointer.

NEXT: The Rock Ethics Institute

Editor's Note: We invite comments and request that they be civil and on-topic. We do not moderate or assume any responsibility for comments, which are owned by the readers who post them. Comments do not represent the views of or Reason Foundation. We reserve the right to delete any comment for any reason at any time. Report abuses.

  1. I’m sceptical about this corpus lingusitcs thingy for the privileged source reason.

    If, in 2050, you looked at written sources in 2019, you would get the impression that in 2019 people thought that a person’s “sex” meant what someone felt about their “gender” identity. Whereas in reality we know that people think a person’s “sex” means whether they have a willy or not. The stuff you’d find written in 2019 newspapers, novels, magazines, blogs; and spouted on the telly would be about what the chatterati would like “sex” to mean.

    Any corpus lingusitcs will privilege the usage of the chatterati over the usage of the hoi polloi.

    1. Of course, the general population will have a better understanding of “sex,” “gender,” and “identity” in 2050 and will recognize that the people who conflated “sex” and “gender” in 2019 were simply ignorant.

      1. Well we can certainly hope that “sex” continues to carry its millenia long willy-related meaning, and that “gender” acquires a usage that settles down to something that remains consistent over at least a whole paragraph.

        But, no, I don’t think the fok of 2050 will be fooled into thinking that the 2019 conflaters of “sex” and “gender” were simply ignorant. There’s much too much evidence that the conflaters were conflating deliberately, so as to try to make it harder to refer to, and hence make arguments dependent on, the concept encapsulated by “sex” in its original meaning.

  2. Stranch has the best of this.

  3. “[Corpus linguistics] draws on the common knowledge of the lay person by showing us the ordinary uses of words in our common language.”

    Except it’s not really about common knowledge. Ultimately words and language more generally are defined not by authority, but by common usage.

    This is why/how language evolves.

  4. This corpus linguistics is a good end to the dictionary fallacy, the fallacy that dictionaries freeze definitions. Language, particularly American English, evolves.

    1. “the fallacy that dictionaries freeze definitions. Language, particularly American English, evolves.”

      All languages evolve, even French, despite the best efforts of the French government to stop it.

      I know of no one who advances this theory you claim is a fallacy. The issue is as I said above common usage defines language (all language) and for most people, the dictionary is the best evidence they have for common usage that goes beyond their immediate peer group.

  5. Not a lawyer here, but instead a professional linguist. I’d agree with the cautions of the dissent to a point, but I’d also note that dictionaries generally only provide denotational meaning, not connotational. Corpus linguistics is a valuable tool for learning what certain terms are applied to. For example, I did a study on what madness meant in early Christianity. Turns out a dictionary definition for the relevant Latin and Greek terms would look not so different from a modern one, but the connotational leaning are extremely different, with the term applied primarily to folks who believed in “heresies” related to the Trinity.

    I know that isn’t from the field of law, but similar situations would apply in legal analysis. This shows why saying “read the dictionary” is not a substitute for corpus analysis. Dictionary definitions are a specific downstream outcome of corpus analytics, not the analysis itself and they necessarily omit exactly the sorts of things that a legal analysis might rely on.

    1. Untermensch, I agree with you. I want to add a bit more. What you say makes sense, or at least it does if you have an expert in relevant context handy, and are willing to engage the expert, and be guided by what the expert tells you.

      Often, that is not how the legal system works. Instead, lawyers and judges tell you that everything which touches on deciding cases gets evaluated according to legal expertise, which they are there to provide.

      I suggest that for lawyers, judges, and law clerks, the corpus linguistics method closely approximates an exercise in automated question begging. On its own—and among legally trained people the corpus will almost always be on its own, because few legal experts are also linguists or historians—the corpus process can never deliver the answer to this question: “Which is the correct context?”

      Instead, the corpus will either deliver all possible examples—and thus obscure the question maximally. Or more likely, it will deliver a subset of possible examples—and thus somewhat obscure the question, but also risk excluding the correct example.

      That possibility—the excluded answer—is one the legal decision maker may stay mindful of. But knowing it is a potential problem does nothing to help evaluate it, instance by instance. To cope with that, there will be workarounds. Whatever non-expert workarounds decision makers hit upon, they will tend to become habitual, and then customary. Eventually, the decision makers and others will come to understand results of those workarounds to be the products of legal expertise—later to be promoted to legal doctrines, akin to those used to guide admissibility of scientific evidence.

      Whether or not the right example is to be found among results delivered by the corpus process, the would-be decision maker is nevertheless still stuck with the problem of deciding which among them features the right context to discern meaning—and whether he has any of the expertise necessary to choose the right alternative from his now-expanded range of choices. He will decide that he does have that expertise—especially now that he has corpus linguistics handy. Thus, practice in researching and presenting corpus results will over time become the legal system’s proprietary linguistics, and its own historiography.

      Alas, nothing about inventing—for specialized service to the legal system—a new linguistics and a new historiography, will substantively alleviate the problem of choosing context and discerning meaning from bygone sources. Those problems will remain as stubborn as before, even after corpus linguistics and its workarounds makes them seem less so. What will happen is that any concerns about bypassing such problems and leaving them unaddressed will be swept out of sight—in favor of misleading assertions that corpus linguistics takes care of them.

      For example, consider what will happen when the corpus in question involves antique and discontinued usages. Some will be usages with which the corpus user is not conversant. Others will be usages, including antique usages, which have not been discontinued, and which approximate or match modern examples with which the corpus user is conversant. An invincible tendency to favor more-familiar alternatives will occur, because that is what always happens when modern non-experts weigh antique meanings.

      Especially if differences seem slight, between an unfamiliar-looking alternative and a familiar-looking one, which non-expert will confidently choose the less-familiar alternative? And on what basis? In short, a bias toward present-minded analysis is a predictable feature of the corpus linguistics method, however it becomes systematized.

      Corpus linguistics will be used in the law, by people trained in law, and not trained in linguistics or history. But the latter two are the disciplines equipped to evaluate unfamiliar, not-present-minded usages—alternatives which will abound among corpus linguistics results. If corpus linguistics could deliver that kind of discernment to people trained only in law, it would indeed be useful, but it would also be a miracle—the equivalent of inventing software to obviate need for professional training to practice linguistics or history—just buy it off the shelf!

      So what will be the realistic expectation if corpus linguistics finds favor in the practice of law? You will get a system which paradoxically frustrates the purpose it is offered to serve—that purpose being insight into unfamiliar meanings derived from obscure bygone contexts. But it will be an automated system, and one loudly purported to deliver higher quality, more systematic results than formerly available. These days, systems which can make promises like those tend to become standards.

      1. Thanks for the thoughtful response. Any fully automated system won’t work. You could couple corpus analytics with technologies like entity and intent recognition or sentiment analysis, but these would be at best tools to find things for further analysis. If anyone claims to fully automate this analysis, it would be the equivalent of fully automating analysis of a legal brief: You might get some useful things from it, but it would be wrong as often as it was right.

        And I think the average lawyer, who is looking for a way to support a point, runs a significant risk of reading meanings into historical content that aren’t there. So your caution is well taken. At the same time, in cases of fundamental indeterminacy, corpus linguistics could help. We already see people trying to do this in 1A law by finding historical references to terms like “militia” and then looking at the contexts. It seems like the results are often rather murky, but it still provides background that would otherwise be lost.

        So corpus linguistics is a tool, and like any tool, it can be applied well or poorly, and the skill of the workman is highly important.

        Incidentally, it’s not surprising that so much of this comes out of Utah. BYU (which appears in a number of the citations) has been a national leader in corpus analytics, and some of the Linguistics Department faculty have served as expert witnesses in trials that hinged on questions of lexicography. Given their background, they introduced these approaches to the courts and made them familiar with them.

  6. It is said, “There are lies, damn lies, and statistics.” The question isn’t whether statistical tools such as corpus linguistics have a place — in law or any other field. The question is how we use it, and what kinds of conclusions we can draw and rely upon. Corpus linguistics will be most useful ‘, or in for obscure terms, such as ’emolument’, or in cases where we suspect that fundamental concepts might have changed over time, e.g., the nature of ‘madness.’ It provides useful evidence; it may shed new light but it can seldom be conclusive in itself.
    But for terms which are more or less ordinary language, such as ‘results in’, general statistics probably should not shift the balance of reasoning very much. Pouring in too many statistics will muddy the flood waters rather than help clear them.

    1. As with anything like this, practitioners need skill. I would not expect a lawyer or judge to get anything useful based on raw statistics. And even in the best case, the analysis component requires considerable care to avoid reading outside conclusions into the results.

      But I would agree absolutely that the results will be too noisy for general language expressions (such as “results in” versus terms of art relevant to a specific field). That’s actually a basic distinction in the linguistic field, and I see pretty major confusion about it in the citations from post.

      So corpus linguistics can be a useful tool if you can restrict the corpus to relevant content and you can identify the tokens that you should analyze. Taking a scattershot approach will lead to problems.

  7. And let’s take another scenario. Lexicographers will omit what they see as marginal meanings or single instances of a meaning. Some dictionaries might include those (like the OED) but most won’t. The power of a corpus is that you might discover that, in a particular year, a meaning was starting to emerge that later lost out. So if you can find the relevant data, you can reveal things that dictionaries, by design, won’t uncover for you. But, again, if you don’t understand how to do that curation and limitation, you’ll drown in irrelevant crap.

  8. As a rule of thumb, this approach would work best in cases where you have independent grounds to believe that your understanding of a term is deficient. I. Other words, if the dictionary definitions provide a clear and compelling case for a reading, getting into the corpus approach seems dangerously like fishing for something that isn’t there. On the other hand, if you read a legal decision and something doesn’t make sense about how a term is used, that may be an indicator that a corpus approach could be useful. But it is definitely something you apply only when the circumstances warrant it and indicate it might be productive.

  9. “When I use a word,” Humpty Dumpty said, in rather a scornful tone, “it means just what I choose it to mean—neither more nor less.” “The question is,” said Alice, “whether you can make words mean so many different things.” “The question is,” said Humpty Dumpty, “which is to be master—that’s all.”

    1. The question is, are you trying to communicate or obfuscate?

Please to post comments