Seemingly Nonexistent Citation in Anthropic Expert's Declaration

The Declaration filed by a "Data Scientist at Anthropic" in Concord Music Group, Inc. v. Anthropic PBC includes this citation:

But the cited article doesn't seem to exist at that citation or at that URL, and Google found no other references to any article by that title.

Bloomberg (Annelise Levy) has a story about this, under the title "Anthropic Expert Accused of Citing Fake Article in AI Lawsuit" (Chat GPT Is Eating the World links to that). Magistrate Judge Susan van Keulen ordered the parties, apparently (according to Bloomberg) referring to this problem, to explain matters:

[N]o later than May 15, 2025, Defendant shall file a Statement addressing the issue raised by Plaintiffs' counsel at the outset of the hearing ….

I'll report what the Statement asserts once it is filed. Thanks to Prof. Edward Lee for the pointer.

Comments

[ah....Clem]

Not too long ago, the dicta was "Don't cite Wikipedia as a source".

Now it seems to be "Don't use AI to generate impressive-sounding but non-existent sources. Or at least be careful to check all of them before submitting to make sure they are real."

[MikeFr]

You would think an Anthropic expert would be aware of AI hallucinations.

I use AI to check contracts, but I always find the cited clause in the original contract. AI in this context is particularly prone to hallucinations. When a clause is similar to a standard clause used in many contracts but different in important ways AI will often hallucinate the standard clause.

[Stephen Lathrop]

I have tried a few times to use AI to evaluate hypothetical historical citations. I judge the failure rate to approach 100% so far.

Problem is, a best-available historical method is often to make historical survivals critique each other. And thus to infer a context useful to understand something which happened in a long-ago past where the original context of creation for the survivals in question has not survived.

That will usually be the case if the history involved is not a recent one. Or if it involves a less-studied record—which is to say a large portion of all the studies likely to deliver new or revised historical insight.

Statistical methods which drive generative AI seem especially unsuited for that kind of analysis. Compared to the entire field of available inferences, the correct answers sought seem almost always to be statistically less likely than a historical counterfactual which some permutation of the training data can support. Distinguishing a less-likely outcome which did happen, from a more-likely outcome which obviously did not happen, seems for the present too challenging for generative AI.

I expect more experience will disclose that to be a commonplace failure mode for generative AI. It will likely plague many different kinds of inquiry, whenever a specific less-likely-but-correct answer is awash in a sea of otherwise commonplace-but-irrelevant statistical possibilities presented by training data.

Whether that is a problem hard-wired into current AI methods, and thus unsolvable without starting anew, seems a question urgently in need of an answer.