Why AI invents case facts, and what actually prevents it

Sanctioned filings get the headlines, but the more common failure is quieter: a demand letter that states a date the record does not support. Here is the mechanism, and the three things that stop it.

The cases that make the news involve invented citations — an attorney files a brief quoting decisions that do not exist, and a judge notices. That is the dramatic version. The version that actually shows up in plaintiff practice is smaller and harder to catch: a demand letter that says the client was hired in March 2019 when the offer letter says May, or that describes a treatment gap of two months when the records show six.

Nobody catches those by reading the letter. You catch them by going back to the file, which is the work the tool was supposed to save.

Why it happens

A language model predicts text that fits its context. Given a partial record and a request for a demand letter, it will produce a fluent, complete-looking demand letter — because that is what fits. Where the record is silent, the model fills the gap with something plausible, and plausible is exactly what a well-written demand contains.

The failure is not that the model lies. It is that it has no way to distinguish "the record establishes this" from "a case like this usually involves this." Both feel identical from inside the generation.

Three things that actually help

These are ordered by how much difference they make, which is not the order most tools implement them in.

1. Give it the whole record, not a summary

Most hallucination in legal drafting is a gap-filling problem. A model working from a 500-word intake summary has to invent everything else. A model that has read the personnel file, the medical records and the text exports has far less to invent, because the answer is in front of it.

This is why "how much of the file does it actually read" matters more than any other question you can ask a vendor. Sampling a long document, or reading only the first N pages, quietly recreates the problem you were trying to solve.

2. Require a citation for every fact

Extract the facts as a separate step, and make each one carry the document and page it came from. Then verify the quote actually appears in that source before keeping it. A fact that cannot be grounded in the text is dropped rather than passed downstream.

This changes the failure mode from "wrong assertion buried in a paragraph" to "missing fact you can see is missing" — which is a far better problem to have.

3. Audit the draft against the record afterwards

Take the finished draft, break it into factual assertions, and check each one against the file. Three outcomes matter, and conflating them is a mistake:

• Supported — the record says this, here is where.
• Contradicted — the record says something incompatible, here is the language.
• Not found — the record is silent. This is not the same as false. It often means the fact came from the client directly, or from a document nobody has uploaded yet.

A tool that reports "not found" as an error will train you to ignore it. A tool that says plainly which of the three it is gives you a list you can actually work through.

What does not help

Asking the model to be careful. Instructions like "do not hallucinate" or "only use facts from the documents" measurably reduce nothing, because the model already believes it is complying. The fix has to be structural — grounding checked outside the generation, not requested inside it.

The practical test

When evaluating any legal AI tool, do this: take a matter where you know the file well. Ask for a chronology. Then pick three facts at random and try to trace each one back to a page. If you cannot, the tool is asking you to trust it about the one thing you cannot afford to trust it about.

Luma is built around that test — every extracted fact carries its source, and drafts can be run back through the record. We would rather you check than take our word for it.