Lawyer's "Repeated Claims That the Spurious Citations Resulted from Clerical Errors Unrelated to the Use of Generative AI Are Not Credible"

From Monday's opinion by Justice Frank Menetrez, joined by Justices Richard Fields and Michael Raphael, in Schlichter v. Kennedy:

Grotke's approach differs from those taken by the attorneys in Noland and Alvarez [two previous cases involving hallucinated citations]. Grotke has not admitted that the Writ and the AOB [Appellant's Opening Brief] contain hallucinated citations that were produced by generative AI. Grotke admitted that he used AI in some fashion when preparing the AOB and that it was "possible" that he used AI in some fashion when preparing the Writ. But he maintains that the four spurious citations resulted from clerical error and that he intended to cite the actually existing cases for the propositions described in the declaration that he filed in response to our order to show cause. We find that Grotke's claims are not credible.

It is difficult to understand how Grotke's four spurious citations could possibly be mere clerical errors, and Grotke has not intelligibly explained how it would be possible. The spurious citations do not involve the mere omission or addition or transposition of one or several digits. Rather, all four spurious citations are completely different from the correct citations for the actually existing cases that have those case names. Grotke's spurious citations bear the hallmarks of hallucinated citations produced by generative AI. "[H]allucinated cases look like real cases. They are identified by a case name, a citation to a reporter, the name of a district or appellate court, and the year of the decision. [Citation.] But, they are not real cases."

Grotke's claim that he intended to cite the actually existing cases is similarly lacking in credibility. The actually existing cases do not support the legal propositions for which Grotke provided the spurious citations in the Writ and the AOB. Consequently, it would make no sense for Grotke to claim that he intended to cite the actually existing cases to support those legal propositions. Grotke attempts to avoid that problem by claiming that he cited the four cases for various other legal propositions, which he describes in his declaration. But the attempt fails, because the legal propositions described in his declaration are not the legal propositions in the Writ and the AOB for which the spurious citations were provided as authority.

For all of these reasons, we conclude that Grotke's repeated claims that the spurious citations resulted from clerical errors unrelated to the use of generative AI are not credible.

Other parts of Grotke's response show a similar lack of candor and credibility. Grotke claimed in his declaration that the spurious citations "resulted from a breakdown in [his] citation-verification process during compilation from vLex." But Grotke admitted at the hearing that before receiving our order of September 19, 2025, he had never signed up for or had a membership on vLex but merely used it "on and off" or "here and there."

Insofar as Grotke claims that he did check the four cases—by searching for them either by case name or by volume and page number citation—before filing the Writ and the AOB, the claim is not credible. If Grotke had tried to check the cases by volume and page number citations, then he would have discovered that the cases do not exist. Grotke admits that is what happened when he searched for the cases in response to our order of September 19, 2025. And if Grotke had tried to check the cases by case names, then he would have discovered that the actually existing cases do not stand for the propositions for which he was citing them.

We agree with Noland and Alvarez that "attorneys must check every citation to make sure the case exists and the citations are correct. [Citation.] Attorneys should not cite cases for legal propositions different from those contained in the cases cited. [Citation.] And attorneys cannot delegate this responsibility to any form of technology; this is the responsibility of a competent attorney." As explained by Alvarez, "[h]onesty in dealing with the courts is of paramount importance, and misleading a judge is, regardless of motives, a serious offense."

For all of the foregoing reasons, we find that Grotke has failed to show cause why he should not be sanctioned for relying on fabricated legal authority in the Writ and the AOB…. [W]e issue a sanction in the amount of $1,750 to be paid by Grotke individually …. We direct the Clerk of this court to notify the State Bar of the sanctions against Grotke.

I e-mailed the lawyer to see if he had a response, and he said this:

The cases were real, not hallucinations, though I have seen AI hallucinate cases in the past.  The cites were just mistaken as to where they were located, page number, volume, etc. I reviewed the cases and included them because they were relevant. I believed that I had the correct cites because they were relevant, but somewhere along the way, maybe AI being the cause, I obtained the wrong cites. As I explained to the court, if I knew exactly why they were incorrect, they would not have been submitted that way.

Comments

[Old Smokin' Egg]

We've seen a number of cases in which lawyers have been fined for including apparent AI hallucinations in their filings. I would hope that those lawyers were not allowed to pass those fines along to their clients; but is there any sort of oversight to ensure that they don't do so?

Furthermore, if a judge discovers hallucinations in a filing and orders the lawyer to amend it, can the lawyer charge the client for the additional hours that it takes to produce that amended version?

[Stupid Government Tricks]

Money is fungible. Fines are an expense. Those fines will be paid for by his clients, and it is almost impossible to prevent him finding some way to bill this client.

[Kenneth Almquist]

I think that the judge is “miffed” that the attorney appears to be lying to the Court, which is a separate offense from using incorrect citations.

I think it is appropriate for the the judge to ask “how did this happen?” before imposing sanctions. It’s hard to imagine a set of circumstances which would justify including multiple incorrect citations in a brief, but my (or the judge’s) lack of imagination doesn’t prove that there are no such circumstances.

[Stupid Government Tricks]

If I were the judge, yes, I'd be curious too. But for the simple goal of discouraging bad citations, it is immaterial how they got there. The attorney blew it. Whether intentionally, from being sloppy or lazy or in a hurry, it just doesn't matter.

Same as buying groceries. If the price of oranges doubles in a week, you might be curious, but whether it was a freeze in Florida, a trucker strike, an oil embargo, or end of season doesn't have much to do with whether you buy fewer oranges or get apples instead.

[David Nieporent]

If I were the judge, yes, I'd be curious too. But for the simple goal of discouraging bad citations, it is immaterial how they got there. The attorney blew it. Whether intentionally, from being sloppy or lazy or in a hurry, it just doesn't matter.

It absolutely does matter, just as it matters whether you ran over your neighbor because you hate him and pressed the gas pedal to the floor when you saw him, or whether you were checking your phone for text messages and didn't see him when he walked in front of your car. Intentional wrongdoing is always worse than negligence.

[David Nieporent]

Sometimes a judge will expressly put in a sanctions order that the money is to be paid by the attorney, not the client. But that's implicit in any sanction directed at an attorney for his own misconduct.

Is there "oversight"? Not in the sense of the judge going over to the lawyer's office and standing over his shoulder while he makes out the sanctions check. But if the attorney tries to bill the client for that (or for the time spent dealing with the sanctions issue), the client can refuse to pay and file a grievance if necessary; that would not go over well for the lawyer.

[Stupid Government Tricks]

I understand the judge being miffed that the lawyer is refusing to say what the judge wants to hear. But the judge seems just as much an ass for insisting the lawyer repeat certain words in the prescribed manner.

The lawyer screwed up. How he screwed up is his problem, not the judge's. Fine him for the bad result, not the manner in which it happened.

[David Nieporent]

See above; that's just wrong. The manner in which something happens is relevant to the punishment imposed for that thing.

[Eric Rasmusen]

Why do judges impose such pitifully small fines? It won't deter. I suppose the embarassment is the big deterrent, but why not make the fine more than symbolic, or, better yet, throw him in jail for 3 days?

[not guilty]

Here the referral to disciplinary authorities based on the finding of lack of candor with the Court is likely the most consequential aspect of the ruling.

[KenveeB]

I've seen attorneys referred to disciplinary orders in a major published opinion and nothing ever comes out of it. Maybe they get a wrist-slap private citation, but nothing published. It just lets the court act like they've Done Something instead of having to actually do anything.

[Darwinnie]

Maybe instead of AI blurbs and pseudonymous cases someone at this blog can post about the “don’t follow illegal orders” message to the military and Trump’s claims those lawmakers are seditious and should be hanged? I’m very interested in the law in this situation. Seems much more important.

[David Nieporent]

1. Service members are obligated not to follow illegal orders.
2. Telling service members they have to obey the law is not sedition.
3. The penalty for sedition is not execution, anyway.

There you go; that's all the relevant law you need.

[Fotheringay-Phipps]

I don't think that's so simple.

It's extremely common - and perhaps even more so in situations involving Donald Trump - that there is disagreement among legal scholars as to whether a given act is legal or illegal. So the question is what standard is required before a military person decides that such-and-such order is illegal and can be disregarded? If dozens of legal commentators and bloggers have announced that it's illegal, does that mean in can be ignored?

Having individual military people make their own legal decisions, even if backed by op-eds declaring things to be illegal, is an extremely dangerous situation.

In light of that context, the question then becomes what these Democrats intended with their call to the military to disregard "illegal" orders. Were they referring to orders that had unanimous consent in the legal community, or currently-valid judicial orders declaring them to be illegal? Or did they mean things that these Democrats and legal people ideologically sympathetic to them declared to be illegal. Donald Trump believes the latter. So do I. I would not advocate shooting these people as traitors because (among other things) I can't PROVE that they meant actions of ambiguous legality, but that's where I differ from Donald Trump. 🙂

[John 4]

Just to be clear, the above is absolutely not legal reasoning. I’m not sure if it’s reasoning in general, but it certainly has nothing to do with the law.

[Rossami]

Since the 'don't obey illegal orders' is a long-standing rule explicitly taught by the service schools and enforced regularly, maybe you should look at the many, many precedents where that exact issue has been raised and decided.

[KenveeB]

You can always start a law blog to post about whatever matters you want covered. Or bring it up in the many Open Thread posts here.

[Darwinnie]

I’m looking for law profs analysis. And the open threads devolve into antisemitism, misogyny, racism, and outright stupidity. I try to maintain some faith in humanity, but I’ll lose it I resort to asking those bozos for their insight.

[Bwaaah]

LOL. An expert opinion including the content you purport to be seeking should be appearing in one or more media outlets over the course of the current news cycle.

I think you understate how much you answer your own questions.

[Incunabulum]

Well, because it's not actually seditious. And because Trump has a lot of bluster for the cameras and nothing will happen because while . . . ill-considered, the admonition to the military is still sufficiently vague to not actually be calling for revolt.

[CJColucci]

There are all sorts of non-AI ways to screw up citations and I have probably done them all, but they were all based on real, existing cases:
1. My notes have Smith v. Jones, 423 F.3d 39 (2d Cir. 2024) and on the line under it Brown v. Green, 378 F.3d 257 (2d Cir. 2024) (all citations in this comment fictitious). My intention is to cite Brown v. Green, which correctly supports the point I'm trying to make, but I cite it as 423 F.3d 39 (2d Cir. 2024);
2. Misreading or transposing digits in citations (I sometimes confuse 4 and 9 because of lousy handwriting);
3. I cite Smith v. Jones, 423 F.3d 39 (2d Cir. 2024) when I meant to cite Brown v. Green, 378 F.3d 257 (2d Cir. 2024).

[MN2828]

And, that is why we had "you better check the citations " beaten into us when we were young lawyers.

[Bored Lawyer]

Inquiry to AI: You gave me hallucinated citations, which I submitted to the Court. The judge is now on my case. Write me a brief that apologizes and avoids sanctions.

[IANNAL]

"Accuracy is paramount."

I've been told that including that phrase in a prompt supposedly reduces hallucinations (at least from ChatGPT), but have not been impressed with the results.

[Bwaaah]

Prompt to ChatGPT: How should I properly use ai-generated legal advice in my presentation to a court?

ChatGPT's answer:

1. Do NOT present AI output as legal advice
2. Verify every factual or legal statement independently
3. Treat AI as a drafting or brainstorming tool, not a source
4. If your jurisdiction requires disclosure, disclose it
5. Maintain full responsibility for the substance
6. Do not attribute intent or consciousness to AI
7. Keep AI output out of evidence unless legally appropriate
8. Use authoritative sources in your actual presentation

Your final version presented to the court should rely exclusively on:
  Statutory text
  Case law you have read
  Rules of evidence/procedure
  Factual evidence
  Rational argument
  Expert input where legally required

AI-generated text should disappear into the background after editing and verification.

[John Michael Thomas]

FWIW I'm an AI consultant.

I don't know if the attorney is lying or not. Here's why:

AI absolutely *WILL* rewrite your fact-checked, fine-tuned, and verified text when it touches it. In fact, it's very difficult to get it to *not* rewrite whatever it feels like rewriting - even if you explicitly tell it not to.

I won't go into the technical reasons why. But this is just the state of the art - it happens on the regular, even with the best AI models and tools.

So, if you have AI generate anything - even if you just want it to proofread and fix your grammar - you must re-verify everything.

The 2-step mitigation:

1. As much as possible, use AI for suggestions, not for writing.

- It's fine to have AI suggest an outline and key points. But don't have it write the actual text.
- It's fine to have AI review your document and suggest changes. But don't ask it to make those changes (make them yourself manually).

2. If you do use AI for writing (at all, in any stage of the process), perform one final, complete human review before you deliver the document.

- If you let AI touch it after you've already reviewed it, you'll waste time, because you'll just have to review it all again anyway.

[Incunabulum]

Whether they are or are not clerical errors is irrelevant. Whatever the source of error the lawyer is ultimately responsible for the product delivered to the court. The errors are the responsibility of the lawyer to manage their resources.

There, IMO, does not need to be an investigation into the source of the error by the court, simply accept that the error exists and discipline IAW with severity of the error and the lawyer's record of performance before the court.

Then let the lawyer sort out fixing the problem themselves.