Review at volume.
Bulk document analysis is where legal AI earns its licence fee. It is also where a confident wrong answer scales to four hundred rows. Here is how to run it so the output is checkable.
01 The economics changed, the obligation did not
Reviewing four hundred agreements for a single provision used to be a staffing question. It is now, mechanically, a fifteen-minute question. That is a genuine change in what a small team can take on, and it is the clearest return in the whole legal AI category.
What did not change: somebody still has to be able to defend the result. The skill has moved from reading everything to designing the question and auditing the answer, and that is a harder skill, not an easier one.
02 Column design is the whole game
A review table's output quality is set almost entirely by how you write the columns. Three principles do most of the work.
Ask for the evidence, not just the conclusion
A column that returns Yes or No gives you nothing to check. A column that returns the answer plus the quoted language it relied on converts every row into something a human can audit in seconds. Always take the quote, even though it makes the table wider.
Split compound questions
"Does this contract permit assignment and if so on what notice?" is two questions, and a model asked a compound question will frequently answer the easier half confidently and skip the other. One question per column, always.
Permit uncertainty
If the only options are Yes and No, you will get Yes or No — including for documents where the answer is genuinely ambiguous or absent. Adding "Unclear" and "Not addressed" as explicit permitted answers does more for accuracy than almost any other single change, because it gives the model somewhere honest to go.
You run a change-of-control column across 400 agreements. 380 return clean answers; 20 return "unclear". Where should a reviewer spend their time?
Decide, then open.
The instinct is the 20 unclear rows, and they do need review — but they are the safe ones, because the tool has already told you to look. The risk lives in the 380.
A defensible allocation: work the 20, then sample perhaps 20–30 of the confident answers with deliberate selection — longest documents, unusual formats, anything from a different drafting tradition. You are not checking whether the tool works. You are looking for a systematic error, one bad assumption repeated 380 times. That is the failure that scales.
03 Where the machine is out of its depth
Some questions are genuinely unsuited to bulk automated review, and knowing them protects you.
| Reliable | Treat with suspicion |
|---|---|
| Explicit, stated facts — dates, parties, notice periods, governing law | Anything requiring the whole agreement to be read as a system |
| Presence or absence of a standard clause type | Whether a clause is market or unusual for this counterparty |
| Extracting defined terms and where they are used | Interaction between provisions in different sections |
| Consistent categorisation against clear criteria | Commercial significance, or what a provision means in practice |
| Flagging documents for human attention | Final decisions with consequences attached |
The pattern: reliable where the answer is in the document, unreliable where the answer requires knowing what normally happens in deals like this.
04 Making it defensible
Assume you will one day have to explain the process to a client, a regulator, or a court. Cheap habits now that make that conversation easy later:
Build the record as you go
- Keep the exact column prompts used, not a paraphrase written afterwards
- Record which model ran the review and when — platforms let you pick, and results differ
- Export the file log for the document set, so the population under review is fixed and provable
- Document the sampling method and its results, including anything it found
- Keep the source-text quotes with the answers, not just the conclusions
- Note who reviewed which exceptions
None of that is burdensome if it is built into the workflow. All of it is nearly impossible to reconstruct months later, which is exactly when it gets asked for.
This week's challenge
Take a review question you would normally phrase as one column and split it into three: the conclusion, the supporting quote, and a confidence rating with "unclear" permitted. Run both versions on the same twenty documents. Compare not the speed but how many rows you can dismiss in under five seconds — that is the number that determines whether volume review is genuinely faster.