The Volokh Conspiracy
Mostly law professors | Sometimes contrarian | Often libertarian | Always independent
Amateur Hour: A Little Bit of Python Can Let ChatGPT Discuss Your Documents
Guest Post on Chat GPT by Professor Seth Chandler (UH)
I am happy to share this guest post from Professor Seth Chandler (UH), who did some pretty cool work with ChatGPT and the Barnett/Blackman 100 Cases book. With this cool new technology, you can ask the authors anything with virtual office hours.
***
We are by now generally familiar with the amazing capabilities of ChatGPT and Large Language Models generally. We may also have confronted an inevitable limitation: during their training, they didn't necessarily see the data you care about most. That limitation, however, is not likely to persist. I want to show what is possible today with a little programming from a Python amateur and what is almost certain to transform legal study and legal practice in a very short time.
Even today, with just a little programming skill, you can take your own documents and use the ChatGPT engine to ask questions based on those documents just as you would on the content with which that engine was originally built. Here's an example. I fed in a 400-page book from Conspirators Randy Barnett and Josh Blackman: "An Introduction to Constitutional Law: 100+ Supreme Court Cases Everyone Should Know." I then asked ChatGPT, "When can a state enforce a generally applicable law that nonetheless adversely affects the religious practices of an individual?" Here's the answer it gave (I asked for 250 words).
"A state can enforce a generally applicable law that adversely affects religious practices of an individual if the law satisfies the test established in Employment Division v. Smith (1990), which holds that if a law is generally applicable and does not target particular religious practices, it need not be justified by a compelling state interest even if it has an incidental burden on religious exercise. Under Smith, a neutral law of general applicability can be enforced even if it impacts religious practices, as long as the government is not singling out a particular religion for discriminatory treatment. However, discrimination targeting specific religious practices is still prohibited under the First Amendment's free exercise clause, as established in the case of Church of the Lukumi Babalu Aye v. City of Hialeah (1993).
The Supreme Court has not provided a bright-line rule to determine when a law is neutral and generally applicable. Rather, the Court has analyzed the language, purpose, and enforcement of the challenged law to determine whether it is aimed at a particular religious practice or is generally applicable. If a law is found not to be neutral or generally applicable, it must satisfy strict scrutiny review and justify the burden on religious exercise by a compelling state interest. In summary, a state can enforce a generally applicable law that adversely affects religious practices of an individual as long as the law is neutral and generally applicable without targeting particular religious practices. If the law is not neutral or generally applicable, then it must satisfy strict scrutiny review and justify the burden on religious exercise by a compelling state interest."
Perfectly respectable. But probably even without ingesting the wisdom of Professors Barnett and Blackman, ChatGPT could have done decently on that softball. Indeed, if all we are considering is text completion, vanilla ChatGPT should be able to compete well with my augmentation version so long as the questions are in areas that ChatGPT already knows about, such as the line of cases and statutes going through Employment Division v. Smith. But then I asked for something more. I asked it for pinpoint sources so that the student could go to the book and read more fully what it said. That's a variant of semantic search that ChatGPT does not presently do. Here's its response.
And now the student can see whether ChatGPT was hallucinating and learn more about the Sherbert to Smith to RFRA to Boerne to Fulton saga. That's a better outcome than ChatGPT alone.
Perhaps that question wasn't challenging enough. Let's give it some facts from a new court case that ChatGPT couldn't have been trained on back in 2021. They come from the website of the Alliance Defending Freedom; my use of them does NOT mean I agree with what is being claimed.
"`The Oregon Department of Human Services (ODHS) requires that people seeking to adopt must "respect, accept, and support" the sexual orientation and/or gender identity of any child who ODHS could place in the applicant's home. This guidance obligates parents to agree to use a child's preferred pronouns, take a child to Pride parades, or facilitate an adolescent's access to medical interventions like puberty blockers and cross-sex hormones. But such rules would force Jessica to violate her religious beliefs. Jessica believes that God created humans as male and female and that a person's God-given sex has spiritual significance for who we are and how we should act. And she believes that a person cannot choose his or her gender because a person's identity is inextricably intertwined with their sex. As a Christian, Jessica always seeks to speak the truth, and as a mother, she desires to raise her children in the faith. Therefore, she cannot say anything to affirm that a person can identify as a gender that is not aligned with the person's biological sex. During her application process, Jessica alerted ODHS that she will gladly love and accept any child, but she cannot say or do anything that goes against her Christian faith. In response, ODHS gave her two choices: abandon your religious convictions or forego the possibility of ever adopting a child. When Jessica stood her ground, ODHS rejected her application for failing to "meet the adoption home standards.' If Jessica sues Oregon, will she win? Tell me the key constitutional doctrines and precedents."
Here's the response from ChatGPT mediated by the text I had it specially ingest. I asked it for less than 500 words.
"The case you describe raises potential issues related to the First Amendment's Free Exercise Clause and the Fourteenth Amendment's Due Process and Equal Protection Clauses. Under the Free Exercise Clause, individuals have a constitutional right to practice their religion and the government cannot prohibit or penalize individuals based on their religious beliefs. In this case, Jessica is arguing that the ODHS is penalizing her by rejecting her adoption application because of her religious beliefs about gender and sexuality. Jessica may argue that the ODHS policy violates her Free Exercise rights because it compels her to express beliefs that conflict with her religious beliefs. However, the Supreme Court has also emphasized that the government can sometimes impose neutral and generally applicable laws on religious individuals or institutions. In Employment Division v. Smith (1990), the Court ruled that the Free Exercise Clause does not provide a right to exemptions from neutral and generally applicable laws. However, some states, such as Oregon, have passed laws that provide greater protection for religious exercise than required by the federal Constitution. Under the Due Process and Equal Protection Clauses, Jessica may argue that the ODHS policy violates her right to parent and raise her children as she sees fit, as well as her right to equal treatment. In Troxel v. Granville (2000), the Supreme Court held that parents have a fundamental right to make decisions concerning the care, custody, and control of their children. In this case, Jessica may argue that the ODHS policy infringes on this fundamental right by requiring her to violate her religious beliefs. Jessica may also argue that the ODHS policy discriminates against her based on her religion. Under the Equal Protection Clause, the government cannot discriminate against individuals based on their religion. Here, Jessica may argue that the ODHS policy targets her because of her religious beliefs about gender and sexuality, which would violate her right to equal treatment.
Ultimately, the outcome of the case would depend on the specific facts and circumstances, as well as how the court applies these constitutional doctrines and precedents."
Good job, although that last sentence is the sort of pablum I would strike out on a student exam. But I like the Troxel citation, the reference to state RFRA's and the general framing of the dispute. And what I also like, again, is the fact that I can go back and see how ChatGPT based its response on the materials I had provided. Here are its top sources.
My point here isn't primarily about ChatGPT's answering capabilities, although even after a few months I continue to be astounded. And it's not really about Constitutional Law either, although I will share at the end of this article some of the other Q&A I had with Professor Barnett and Blackman's book. It's about how little you need to know right now to do exactly what I did. And what I did is the rough equivalent – sorry Randy and Josh – to having office hours with the author of a constitutional treatise.
As long as you can get the materials in some reasonable digital format, law faculty law students, the average lawyer, indeed almost anyone with modest computer skills, can slurp text into a computer, split it up into reasonable chunks, send the chunks off to OpenAI or others' "embedders" that convert each of them into a vector of numbers, and then send these "embedding vectors," along with some metadata (like the page numbers or underlying text) into what is known as a "vector database" such as Pinecone that is optimized for exactly the process described here. The cost of "semantic search" has been reduced by many orders of magnitude. Once I had my Python program written (see discussion below), the whole process for the 400-page book took about a minute and cost me about 10 cents – OpenAI charges a small amount to embed text. And while, yes, today, the process I described did take some computer savvy, because I'm thoughtfully sharing the code, you can now do it too. Also, lest you be overly impressed, ChatGPT was a huge help along the way. One thing that neural network does really, really well is to write, adapt and debug Python code.
And what I did is amateur. Although I am a 33-year veteran of Wolfram Language (Ok, so I'm not a newbie to computer science), my knowledge of Python is genuinely limited. Others will not be challenged. Within a few months, there is surely going to be an app in which you just tell the computer what files to ingest, wait a few minutes, and receive a link to plugin to which you can point ChatGPT. Once you have that plugin, you can then commence an endless fireside chat with the corpus you uploaded. Indeed, it doesn't have to be a single chat. You could have multiple corpuses loaded and see how the two of them interact with each other. Imagine a synthetic dialog between GPT-Williston and GPT-Corbin on contract law or GPT-Scalia and GPT-Kagan battling over the nature of substantive due process. (At least the discussions would be well written!).
Yes, yes, there are issues. How are students going to learn anything when they have perpetual office hours and can just ask the machine? What about irksome intellectual property issues. I cleverly avoided the wrath of these authors by not returning full text from the sources. I just provide pointers that let owners – including new owners -- of the underlying material search it with pinpoint accuracy. Surely, as is always the case, the pioneers in this area will be the porn industry. And who knows what profits the owners of digital rights will charge for permitting strong semantic search on materials otherwise difficult to get in digital form. It might make the cost of today's casebook supplement look like a pittance.
In the meantime, however, rejoice and think of positive use cases. I'm thinking of abolishing office hours and just uploading transcripts of my classes and exports from the PowerPoints right into ChatGPT. (Just kidding, dean). Or take every brief from some organization and see if AI can find inconsistencies. Or, if I were a practitioner, stop paying a fancy vendor and just create a DIY semantic search engine to pour over my opponent's document production.
Who knows where we are heading? I rather suspect this column will seem quaint in a very short time and that I may be expressing undue excitement about the fax machine of generative AI. But if the lack of serious impediments in my foray is representative, wherever we are headed, we are going to get there very fast.
Appendix
Here are some other questions and answers.
Question: "According to South Dakota v. Dole, are there any limits on Congress's power to place conditions on spending? In your answer, please cite the most relevant cases." [300 words requested]
Answer: "Yes, there are limits on Congress's power to place conditions on spending. The Supreme Court case South Dakota v. Dole (1987) identified five limitations on Congress's spending power. First, the exercise of the spending power must be in pursuit of the general welfare. Second, Congress must place conditions on the funds unambiguously. Third, the conditions must relate to the federal interest for which the spending program was established. Fourth, other constitutional provisions may provide an independent bar to the conditional grant of federal funds. And fifth and finally, a condition becomes unconstitutional when the financial inducement offered by Congress might be so coercive as to pass the point at which pressure turns into compulsion. This coercion would, in effect, commandeer the state legislature to comply with the condition. In addition to these limitations, the Court has recognized limits on Congress's power to attach strings to money given to states due to federalism concerns. Justice O'Connor's dissent in South Dakota v. Dole provided a more narrow test for "relatedness" or "germaneness." Later cases, including New York v. United States (1992) and Printz v. United States (1997), have held that Congress lacks the power to direct or commandeer states to take certain actions. In NFIB v. Sebelius (2012), the Supreme Court finally identified a condition on spending as coercive, recognizing this as an unconstitutional limitation on Congress's power. Therefore, while Congress has broad discretion to spend money, it cannot attach conditions that exceed the scope of its spending power or unduly coerce states."
Sources:
Question: "Under what circumstances would it make a difference in a plaintiff was seeking relief under the Privileges and Immunities clause of Article IV or under the dormant commerce clause?" [100 words]
Answer: "It would make a difference in what specific right the plaintiff is seeking to vindicate. If the plaintiff is seeking relief under the Privileges and Immunities Clause of Article IV, they are likely seeking protection of fundamental rights that are shared by all US citizens. On the other hand, if the plaintiff is seeking relief under the dormant commerce clause, they are likely arguing that a state law discriminates against interstate commerce in violation of the Commerce Clause. While there may be some overlap between the types of rights protected by these two clauses, they are conceptually distinct and can result in different outcomes in litigation."
Sources:
