The Volokh Conspiracy

In 2011, Apple introduced Siri. This voice recognition system was designed as an ever-present digital assistant, that could help you with anything, anytime, anywhere. In 2014, Amazon introduced Alexa, which was designed to serve a similar purpose. Nearly a decade later, neither product has ever reached its potential. They are mostly niche tools that are used for very discrete purposes. Today's New York Times explains how Siri, Alexa, as well as Google Assistant lost the A.I. race to tools like GPT. Now, we have another notch in the belt of OpenAI's groundbreaking technology.

Yesterday, OpenAI released GPT-4. To demonstrate how powerful this tool is, the company allowed a number of experts to take the system for a spin. In the legal corner were Daniel Martin Katz, Mike Bommarito, Shang Gao, and Pablo Arredondo. In January 2023, Katz and Bommarito studied whether GPT-3.5 could pass the bar. At that time, the AI tech achieved an overall accuracy rate of about 50%.

In their paper, the authors concluded that GPT-4 may pass the bar "within the next 0-18 months." The low-end of their estimate proved to be accurate.

Fast-forward to today. Beware the Ides of March. Katz, Bommarito, Gao, and Arredondo posted a new paper to SSRN, titled "GPT-4 Passes the Bar Exam." Here is the abstract:

In this paper, we experimentally evaluate the zero-shot performance of a preliminary version of GPT-4 against prior generations of GPT on the entire Uniform Bar Examination (UBE), including not only the multiple-choice Multistate Bar Examination (MBE), but also the open-ended Multistate Essay Exam (MEE) and Multistate Performance Test (MPT) components. On the MBE, GPT-4 significantly outperforms both human test-takers and prior models, demonstrating a 26% increase over ChatGPT and beating humans in five of seven subject areas. On the MEE and MPT, which have not previously been evaluated by scholars, GPT-4 scores an average of 4.2/6.0 as compared to much lower scores for ChatGPT. Graded across the UBE components, in the manner in which a human tast-taker would be, GPT-4 scores approximately 297 points, significantly in excess of the passing threshold for all UBE jurisdictions. These findings document not just the rapid and remarkable advance of large language model performance generally, but also the potential for such models to support the delivery of legal services in society.

Figure 1 puts this revolution in stark contrast:

Two months ago, an earlier version of GPT was at the 50% mark. Now, GPT-4 exceeded the 75% mark, and exceeds the student average performance nationwide. GPT-4 would place in the 90th percentile of bar takers nationwide!

And GPT scored well across the board. Evidence is north of 85%, and GPT-4 scored nearly 70% in ConLaw!

We should all think very carefully how this tool will affect the future of legal services, and what we are teaching to our students.

A new paper, by Daniel Martin Katz, Michael James Bonnarito, Shang Gao, and Pablo Arrendondo. posted here on SSRN, report on the outstanding performance of ChatGPT-4 on the Bar Exam (Multistate Essay and Multistate Performance).  Congratulations, ChatGPT!

From the Abstract:

In this paper, we experimentally evaluate the zero-shot performance of a preliminary version of GPT-4 against prior generations of GPT on the entire Uniform Bar Examination (UBE), including not only the multiple-choice Multistate Bar Examination (MBE), but also the open-ended Multistate Essay Exam (MEE) and Multistate Performance Test (MPT) components. On the MBE, GPT-4 significantly outperforms both human test-takers and prior models, demonstrating a 26% increase over ChatGPT and beating humans in five of seven subject areas. On the MEE and MPT, which have not previously been evaluated by scholars, GPT-4 scores an average of 4.2/6.0 as compared to much lower scores for ChatGPT. Graded across the UBE components, in the manner in which a human tast-taker would be, GPT-4 scores approximately 297 points, significantly in excess of the passing threshold for all UBE jurisdictions. These findings document not just the rapid and remarkable advance of large language model performance generally, but also the potential for such models to support the delivery of legal services in society.

The DEI Dean at Stanford Law did not think Judge Duncan's juice was worth the squeeze. I pose a related question: are all of Stanford's graduates worth the squeeze? Without question, these students have excellent academic credentials, and performed very well on various forms of testing. But at least some of them exercised terrible judgment. They decided that it was appropriate to shout down and heckle a sitting federal judge. They yelled vulgar slurs about parts of the human anatomy. Now, to be fair, these students have been inculcated in DEIdeology. They were taught by Dean Steinbach and others that this sort of behavior is not only appropriate, but is a necessary response to a harmful presence on campus. So perhaps the students should not bear all of the blame here. Stanford is at least responsible in part; plus their undergraduate institutions; and their K-12 teachers (current 1Ls were born circa 2000); don't forget coaches that handed out participation prize trophies; and so on.

Thankfully, Stanford can help fix this mess. To paraphrase Dean Steinbach, Stanford can separate the juice from the pulp. But how? The first step would be to identify the students who violated the code of conduct. The joint statement from the President and the Dean seems to suggest that at least some students crossed the line. The event was recorded from multiple camera angles. It would not require Zapruder-level scrutiny to figure out who was at fault. Now, Stanford may not really want to identify the assailants. We never did find out who placed black tape on the faculty photos at Harvard Law School.

I do find it telling that the protestors stood outside Dean Martinez's class while wearing custom-printed-masks. These non-N-95 masks did not provide much COVID protection, but were very effective at concealing identifies. I heard from a colleague that the masks would allow more students to participate who feared potential repercussions.

After the relevant students are identified, Stanford would have several options. I'll list them in order of increasing severity. First, the college could simply issue a warning to those who violated the code of conduct. There would be a recognition that they transgressed, but no consequences at the moment. However, if they were to violate the code again, there would be some distinct consequences. A one-bite rule, so to speak, would perversely incentivize every student to have one, and only one, moment in the sun. I imagine that students who are unfortunate enough to get a mere warning would let all other students engage in future risky conduct.

Second, Stanford could impose some form of extracurricular punishment. For example, students cannot participate in organizations like moot court or law journals, or serve as research assistants for professors. These consequences would deprive elite students of that which they hold most dear--credentials for their resume and recommenders for clerkships. And, in a way, these sanctions would signal to prospective employers that something was amiss. A student was on law review for 2L, but not 3L? Still, there would be no official imprimatur of misconduct.

Third, Stanford could issue such an imprimatur. The college could convey a message to the character & fitness review board of the relevant state. Now, in California, asking Judge Duncan about the female anatomy may warrant some sort of special commendation from the health law section of the state bar. 51 imperfect solutions, and all. But for other, more sane jurisdictions, this sort of conduct should have repercussions. Still, these communications to the bar are not (as far as I know) publicized.

Fourth, Stanford could make things public. It could issue a press release naming and shaming all of the students who violated the code of conduct. Presumably, any law firm or judge who hired or is planning to hire a SLS grad would check this list, and may proceed accordingly. Judges Ho and Branch offer a similar recommendation in National Review.

And if schools are unwilling to impose consequences themselves, at a minimum they should identify the disrupters so that future employers know who they are hiring.

Schools issue grades and graduation honors to help employers separate wheat from chaff. Likewise, schools should inform employers if they're injecting potentially disruptive forces into their organizations.

This move would send a signal to all of Palo Alto, and indeed, the entire country: there are actual consequences for such behavior. Then again, some firms may want to hire these students. Good for them! Hopefully they do not exercise such behavior in the John Minor Wisdom Courthouse. Some judges may also view such conduct as admirable. Professor Rory Little wrote that heckling is a new form of "principled civil disobedience"! Good for them! If judges accept this worldview, we're in big trouble.

Fifth, there is the option of expulsion. These students can forget about clerkships or summer associate gigs. They will not even be allowed to sit for the bar. Still, I think this option would be draconian, especially for a first-time offense.

Stanford has options. Let's see if it does anything about this situation, or if it will simply issue naked apologies.

Nature, last month (paywall):

The US Food and Drug Administration (FDA) will soon require [based on new federal legislation] researchers and companies seeking approval for late-stage clinical trials to submit a plan for ensuring diversity among trial participants…. The diversity requirement arrives in the wake of a 2022 report from the US National Academies of Sciences, Engineering and Medicine, which found that, although the representation of white women in clinical trials has improved, progress has "largely stalled" for minority racial and ethnic groups.

New York Times, yesterday:

Grappling with the deep history of racism in Western science, the National Academies of Science on Tuesday released guidelines recommending that scientists not use race as a category in genetic studies.

The guidelines, produced in response to a directive from the National Institutes of Health, noted that racial categories were poor proxies for genetic diversity and that social and environmental factors, like poverty and injustice, were often overlooked.

One possible distinction between these scenarios is that while everyone might agree that "race" is not a good proxy for genetics, it's important to have "representation" of minority groups in late-stage clinical trials so that so members of these groups have confidence in the results, while there is no similar concern in genetic studies. However, as I've pointed out before, there is no reason Americans should think that "Hispanic" or "Black" or "Asian" are medically significant classifications, but not much genetically "narrower" (because historically more genetically isolated) groups like "Ashkenazic Jews" or "Icelanders" unless the authorities told them to be worried about the former categories.

Indeed, even if one thinks that "race" correlates sufficiently with genetics to be medically significant, "Hispanics" can be of any race or any combination thereof, "Asian American" includes Austronesians, Caucasians, and East Asians, and "Black" Americans have a large admixture of European ancestry. In other words, the standard American "racial" categories that were adopted for entirely different purposes but are used by default in medical and scientific research are at odds with what racialists consider to be "race."

For more on related matters, including how FDA and NIH came to require many biomedical companies to use crude "racial" classifications to begin with, see chapter 6 of Classified.

In a new post, Sander recounts the history of his work on "mismatch" (the incoming academic qualifications gap between students who are favored in admission because they contribute to 'diversity' and the student body as a whole) in law schools. After noting that it's been extremely difficult to acquire additional data,

Robert Steinbuch (a colleague at the University of Arkansas, Little Rock) and I eventually secured the public release of data from 12 cohorts of law students at four law schools, covering about 6,500 students in all. And after a multi-year review process, the Journal of Legal Education—the official organ of the Association of American Law Schools—has now agreed to publish the first set of our results in its next issue.

Our findings indicate that mismatch can account for two-thirds to three-quarters of the Black-white gap.

Our findings are even stronger than we expected. A student's degree of mismatch in law school is by far the strongest predictor of whether he or she will pass a bar exam on a first attempt. In 2005, I estimated that mismatch could account for half of the bar-passage gap between Blacks and whites, with the rest caused by lower average pre-law-school preparation levels among Black students. Our findings indicate, however, that mismatch can account for two-thirds to three-quarters of the Black-white gap, as well as more than half of the Hispanic-white gap. Importantly, we found that when one fully controls for mismatch, LSAT, and UGPA, race-specific effects completely disappear. Many critics of mismatch had argued, without evidence, that poor minority performance might be caused by "hostile" environments in law school or "stereotype threats" faced by underrepresented students. Our results flatly contradict such claims.

Most of our results are in regression analyses that can be hard for those without a technical background to interpret. But one of our tables, reproduced below, makes the basic pattern clear. It shows first-time bar-passage rates for several thousand law graduates, grouped by their LSAT score and whether they attended an elite, somewhat-elite, or non-elite law school. Unsurprisingly, students with high LSAT scores had high bar-passage rates at all schools and did particularly well at the elite school in question. But students at the elite school with LSAT scores 12 to 14 points below the median of their fellow students (i.e., LSAT scores of 150-152) had only a 22 percent first-time bar-passage rate, while students with the same LSAT score at the non-elite school in question had a 79 percent first-time bar-passage rate. In other words, lower mismatch translates into dramatically better performance.

I wrote an oped on a related subject almost twenty years ago, and I heard from several law school deans and administrators (none of whom wanted their names or schools identified, naturally). Each of them told me that there was a "cutoff," known at their law school, below which the odds of student bar passage plummet. You can see this effect in the mid-150s at UCLA, the low 150s at UC Davis, and the mid-140s at UA Little Rock.

They added that if they resisted accepting URM students with LSATs below that threshold, the ABA accreditation people threatened them with being placed on probation for having an insufficiently diverse student body--even though ABA rules required law schools not to admit students that they thought would not succeed academically.

In other words, the ABA prohibited them from taking white or Asian students with LSATs below a certain threshold, knowing that those students were unlikely to become lawyers, but required them to admit at least some Black or Hispanic students with those scores. Worse yet, no one informed the students admitted with those LSATs that their odds of ultimately passing the bar were low. (Indeed, Sander's chart understates the matter, because it excludes students who fail out or drop out of law school before they have a chance to take the bar.) Even worse, at a public hearing I participated in, an ABA official acknowledged that for accreditation purposes, the ABA only cares how many URM students matriculate, and does not care, or track, how many ultimately make it through law school and pass the bar!

[Earlier posts in this series: Introduction/#2/#3/#4/#5]

The following is an excerpt from Stefan Zweig's "The World Of Yesterday," his magnificent tribute to life in the Austro-Hungarian empire in the first several decades of the 20th Century (and the utter destruction of that world in the aftermath of World War I), describing the hyper-inflation of the 1920s in Austria and Germany:

An economist who knew how to describe graphically all the phases of the inflation which spread from Austria to Germany, would find it unsurpassed material for an exciting novel, for the chaos took on ever more fantastic forms. Soon nobody knew what any article was worth. Prices shot up at random: a box of matches could cost twenty times more in a shop that had raised the price early than in another, where a shopkeeper was still selling his wares at yesterday's prices.… Standards and values disappeared during this melting and evaporation of money. There was but one merit: to be clever, shrewd, unscrupulous, and to mount the racing horse instead of be trampled by it…. The most grotesque discrepancy developed with respect to rents, the government having forbidden any rise; thus tenants, the great majority, were protected but property owners were the losers. Before long, a medium-size apartment in Austria cost its tenant less for the whole year than a single dinner….

The strangest thing is that I cannot recall, however I may try, how we kept house during that era, or in what manner the Austrians kept on raising the thousands and tens of thousands - and the Germans, in their turn, the millions - of kronen which were dally needed to keep body and soul together. Mysteriously enough, they did raise them. Habits are acquired and the chaos became normal to life. It stands to reason that one who was not a witness would imagine that, at a time when an egg cost what a fine motor-car used to cost (in Germany eggs went up to four billion marks, the approximate past value of all of the real estate in Greater Berlin), women must have been running wildly through the streets with tousled hair, that shops were deserted for lack of purchasing power and that theatres and amusement places were surely empty.

Astonishingly enough, just the opposite was the case. The will to pursue life was great enough to overcome the instability of the currency. Financial chaos prevailed, yet the daily round seemed little affected. There were widespread individual changes, such as those who had wealth in the form of cash in bank or government bonds becoming impoverished, speculators becoming rich. But the balance-wheel maintained its rhythm unconcerned with single fates, there was no standstill; bakers baked bread, cobblers made boots, authors wrote books, peasants sowed and reaped, trains ran on time, the morning newspaper never failed, and the places of entertainment, bars, and theatres were filled to capacity. The very fact that what once represented the greatest stability —money— was dwindling in value daily caused people to assess the true values of life-work, love, friendships, art, and Nature the more highly, and the whole nation lived more intensively and more buoyantly than ever despite the catastrophe; young people went on mountain tramps and returned healthily tanned, dance halls kept going until late at night, new factories and business enterprises sprang up. I don't think that I ever lived and worked with greater zest than in those years. Whatever had meant much to us in days gone by meant even more now; at no time had we ever been so devoted to art in Austria as in those years of chaos, because the collapse of money made us feel that nothing was enduring except the eternal within ourselves.

In October, the Supreme Court held several hours of oral argument in the Students for Fair Admissions case, challenging the constitutionality of affirmative action preferences for African Americans, Hispanics, and Native Americans at Harvard and the University of North Carolina.

The questions covered a wide range of issues, including, I believe for the first time, significant questions from the Justices about coherence of the classifications used by the universities.

The key issue in the cases is whether these classifications can be used by the universities to ensure racial diversity. The phrase "racial diversity" came up in oral argument thirty-eight times, with Justices Kagan and Sotomayor in particular pressing counsel on the issue.

The question no Justice asked is why the universities treat "Hispanic" as a "racial" classification. The Department of Education, consistent with OMB regulations that apply across the federal government, treats Hispanic as an "ethnic" classification; Hispanics can be of any race. The Common App, used by Harvard and UNC, follows the federal convention in asking students first whether they are Hispanic, and then about their race:

Are you Hispanic or Latino/a/x?

YesNo

Which best describes your Hispanic or Latino/a/x background? (You may select one or more)

Central America

Cuba

Mexico

Puerto Rico

South America

Spain

Other

In any event, despite Justice Sotomayor's protestations in her BAMN dissent that "race matters" with regard to Hispanics (but apparently not Asian Americans) in education, treating Hispanics as a race seems to violate federal law. (I have previously discussed how the Supreme Court wound up treating Hispanic as a racial classification, even though that's legally incorrect.) For example, I don't think that a university could legally prefer Greek American applicants over other applicants in the name of "racial diversity." Discrimination of this sort within the "white" classification would generally be considered illegal ethnic/national origin discrimination.

I expect if counsel for the universities had been asked how they can defend, contrary to federal law, treating "Hispanic" as a race, the first response would be stunned silence, because universities have never even thought about that issue. Once counsel recovered, the answer would be that the universities just use "racial diversity" as a shorthand for pursuing racial and ethnic diversity, and the latter includes Hispanics, including those who check the "white" box.

But that answer would raise serious constitutional problems of its own. If the universities are interested in ethnic diversity, why is Hispanic literally the only ethnic group that they keep track of to ensure such diversity? One could argue that Hispanic is a special classification, because many Hispanics are dark-complexioned, and thus have experiences not common to other ethnic groups. But then what of, e.g., Arab Americans, Armenian Americans, Persian Americans, or Berber Americans? Hispanics, one might argue, face discrimination that other ethnic groups do not. But can one plausibly argue that Arab and Iranian Americans, especially those who are identifiably Muslim, have not faced discrimination not common to those with whom they share the "white" classification?

The heart of the equal protection clause, which governs this case (including for Harvard, because that clause has been deemed coextensive with Title VI of the 1964 Civil Rights Act) has long been a prohibition on arbitrary classifications. An applicant who checks off "Hispanic," then "White," then "South America" who is of Italian-Argentine descent gets a racial diversity preference. An applicant who checks off "not Hispanic," then white, then Middle Eastern who is an Iraqi Yazidi survivor of genocide does not. Nor does a very-dark complexioned Egyptian Copt, a member of a group that has faced centuries of caste-like discrimination in their home country. Arbitrariness, anyone?