Nine questions to ask any legal AI vendor

Most demos are designed to prevent these questions from coming up. Ask them anyway — the answers separate tools that read your file from tools that autocomplete a template.

Legal AI demos are unusually good at looking impressive, because the hard parts are invisible. A polished demand letter tells you nothing about whether the tool read page 1,400 of the medical file. These questions surface what the demo hides.

1. How much of a long document do you actually read?

Ask specifically about a 2,000-page file. Some tools sample. Some read the first N pages. Some summarise and then work from the summary, which loses exactly the detail you are paying for. The answer should be "all of it", and they should be able to explain how.

2. Can I trace any given fact back to a page?

Not "does it cite sources" — every vendor says yes. Ask them to click a fact in the demo and show you the page. Then ask them to do it with a fact you choose.

3. What happens when two documents disagree?

The right answer is that both are surfaced and the conflict is flagged. A tool that silently picks one has made a legal judgment on your behalf, without telling you it did.

4. Do you handle scans, handwriting, audio and video?

Real case files contain phone photos of documents, recorded calls, and video. If the tool only handles clean digital PDFs, someone at your firm is still doing the hard part.

5. Is our data used to train models?

Get this in writing, and ask specifically about their model providers, not just about them. "We do not train on your data" is compatible with a sub-processor that does.

6. What actually happens when we delete something?

Deleting a document should remove what was derived from it — extracted facts, embeddings, cached text. Ask directly, because "deleted" often means "hidden from the list".

7. Who can see which matters?

If the tool cannot restrict a paralegal to their assigned matters, it will not survive a conflicts discussion.

8. What does it cost when we are busy?

Per-seat pricing punishes you for putting the whole firm on it. Usage pricing with an unclear meter punishes you for a heavy month. Ask what a bad month looks like on the bill.

9. Can we try it on our own file before we commit?

The answer should be yes, immediately, without a procurement process. Any hesitation here is worth paying attention to.

For what it is worth, we would rather you asked us all nine than watched a polished demo. The messiest file you have is the only test that tells you anything.