When ChatGPT first hit the mainstream in 2022, I realized that law school assessments would need to change. At the time, I asked my ConLaw students what they thought about oral exams, like in ancient times. Their immediate reaction was one of absolute panic and dread. At the time, I didn't pursue the matter further. My exams were proctored in class, and used locked-down laptops, so I didn't think it was feasible for students to use AI during the proctoring.
Four years later, things have changed. I've watched many YouTube videos of how students expertly cheat on exams using AI. They place vaseline or some other agent in front of their webcam, take pictures of the question with their phones or smart glasses, and provide perfect answers. The situation is even worse with any sort of take-home work. The temptation for students to provide perfect answers every time is too high. Indeed, a student who doesn't use AI is putting herself at a disadvantage.
Some professors have responded to this trend by embracing AI. They encourage their students to use it "responsibly" (whatever that means) and to show their prompts and other searches. I, for one, remain an AI luddite. I assure all of my readers that every word I write comes from my brain, without the use of generative AI. Generally, I write a post straight through without pauses, and then go back to clean it up. (My typos are proof that I don't use AI.) Unfortunately, Google now pushes AI to the top of every query, though I always click through the links and check primary sources. I never trust the Gemini summary.
I would highly recommend a memo prepared by Dean Bobby Chesney at the University of Texas. Chesney relays a number of issues his faculty has considered concerning AI. In particular, Chesney writes that "we've also seen a surge of interest in assessments involving class participation and live presentations (even oral exams in some cases)."
I have come to much the same conclusion. My tentative plan for ConLaw in the spring is to switch to an oral, in-person midterm.
Taking a step back, I have long struggled with how to use midterms effectively. The most useful aspect of a midterm is to provide an early intervention--feedback for students who need help. But by the time the midterm is graded, it is often too late in the semester to correct course. (I do not give multiple choice questions in ConLaw, though that format would make it far easier to grade quickly.) When I started teaching, I would proctor the midterm right before spring break, and then spend the entire spring break grading midterms, and return them when we got back. Needless to say, with family commitments, that is no longer an option. The other problem with midterms is that some students have not yet hit their stride by the seventh week. Indeed, there was not always a clear correlation between grades on the midterm and grades on the final.
All of these concerns remain. Still, in light of the threat to assessments posed by AI, I will need to redouble my efforts. Here is my working approach: halfway through the semester, I will proctor an individualized, oral midterm in my office. Each session will last approximately five minutes. If I do ten sessions per hour, with breaks, I can wrap up the entire process in a single day. There is an advantage to doing everything in a short period of time to reduce the risk of students talking to each other. At my law school, 1Ls do not have classes on Friday, so that timing would work well. In the first instance, I would randomly assign students to slots throughout the day. If they have an actual, documented conflict, they can reschedule. And there will be two weeks of advance notice, which should help reduced potential conflicts.
I will need to give some more thought about how to handle timing for accommodated students. As most professors know, the number of students who have testing accommodations has skyrocketed in recent years. Often accommodations are provided for students who are not able to write with a pen or type with a keyboard--so-called "scribing." But the oral exam eliminates that need. Students would only need to listen and speak.
Students would not be allowed any materials or devices during the sessions. I'll leave a basket (maybe a Farraday box) outside my office to deposit all tech. Some smart glasses can be disguised as regular glasses. I hope this doesn't become an issue.
I would also record and digitally transcribe the entire session, should any disputes arise later. The midterm would not be worth a large portion of the grade. I think all law schools permit professors to vary grades a small amount based on participation. I would make that participation score turn on the performance of the in-person midterm. (I've never figured out how to accurately and consistently record class participation over the course of a semester, so I don't try.)
The substance of the midterm will simulate a final exam, but on a reduced scale. I will circulate five fact patterns to the student in advance. That way students can prepare and will not be surprised. The questions will not be a full-blown fact pattern like the final, but a single paragraph that has embedded a few related ConLaw issues. I will tell students that during the oral midterm, they will pick one of the questions at random, maybe by drawing an index card from a deck.
Students will then have to give a one minute prepared remark answering the prompt--basically the opening statement a lawyer would give during an oral argument. At that point, I can ask a series of follow-up questions to probe different aspects of the answer. I might ask them about a case they mentioned, or ask why they didn't mention a particular case. This would be the sort of Socratic Dialogue I do in class every week. Indeed, I imagine I would use the same questions I pose in class, so there will be fewer surprises.
At the end of the session, I would provide feedback on the spot. I imagine the grade will be a number between 1 and 5, or 1 and 10. I can then point out where they gained and lost points. This feedback would be immediate and personalized. Students usually receive grades on midterms weeks or even a month later, long after they've forgotten what the question was.
Students would be instructed to not talk with their classmates about the exam. I am skeptical of how closely this instruction will be followed. The upshot of providing the questions in advance is that students can work together, or with their Chatbot, to prepare answers. But all of that preparation goes out the window once I start asking followup questions.
Using an oral midterm is sort of an interim solution. The ultimate move would be to make the final exam an in-person oral exam. A challenge is anonymity. While final exams require anonymity, it is less important for midterms. Professors are expected to go over a student's midterm to provide one-on-one feedback. But perhaps as so much of the work students submit is no longer their own, but instead created by AI, eliminating the cloak of anonymity may be the final way to determine what a person actually knows.
As I said above this plan is tentative. (Then again, a blog post I wrote years ago concerning "tentative" thoughts is still being cited as my definitive view.) I welcome thoughts and comments from other professors who have considered these issues.
