When AI and Human Reviewers Disagree, Who Is Actually Wrong?

As generative AI becomes a more meaningful part of document review, much of the conversation has centered on a fairly obvious question: can we trust AI to make accurate decisions? That is certainly an important question, particularly when technology is being used to identify responsive material or reduce the population requiring traditional eyes-on review.

There is, however, another question that deserves just as much attention. When an AI-assisted review result is compared against a completed human review, should we automatically assume that the human coding represents the correct answer?

That assumption is built into many validation exercises. Human coding is typically treated as the benchmark, and any disagreement between the technology and the original reviewer is counted as an AI error.

The problem is that human review has never been perfectly consistent. Treating historical coding as an infallible answer key can materially distort how AI performance is measured and, in turn, how legal teams assess whether a technology-assisted workflow is reliable enough to use in practice.

A recent proof of concept conducted by Lineal in tandem with a pharmaceutical company illustrated that point particularly well. We applied Relativity’s aiR Review to a previously reviewed document population and compared the AI’s relevance determinations against coding from a review that had been completed by a third-party review provider, not by Lineal.

Starting With a Known Review Population

The proof of concept began with 28,837 documents from the completed review. aiR Review successfully analyzed 23,866 of those documents, while 4,971 could not be processed and were excluded from the AI analysis. Lineal worked with the legal team to develop and refine the review prompt, ran aiR Review across the available population, and then compared those results to the original third-party reviewer coding.

Using the prior human review as the benchmark, the initial results showed 86.38% precision, 75.11% recall and 78.58% overall accuracy. Those figures suggested strong performance in some areas, but the recall number warranted closer examination because of the apparent size of the false-negative population.

There were 3,472 documents that aiR Review classified as Not Responsive even though the previous human review had coded them Responsive. Under a conventional benchmark analysis, each of those documents would be recorded as a false negative and attributed to the AI unless the underlying disagreement was examined more closely.

Looking Behind the Disagreement

False negatives are understandably one of the principal concerns when evaluating any technology-assisted review methodology. Over-inclusion generally creates additional review work, while under-inclusion raises the more consequential concern that potentially relevant information could be missed.

Rather than simply accepting all 3,472 disagreements as AI failures, Lineal selected a sample of 253 documents from that false-negative population and independently re-reviewed them against the matter’s responsiveness criteria. The purpose was to determine whether the AI had actually missed responsive documents or whether some of the apparent errors resulted from inconsistencies in the historical coding.

Of the 253 documents reviewed, only 11 were confirmed as true AI misses, or approximately 4.3% of the sample. The remainder of the sampled documents did not support the original responsiveness coding when independently re-reviewed against the applicable criteria.

That observed rate can be used to estimate what the broader disagreement population might look like. If the roughly 4.3% true-miss rate found in the sample held across all 3,472 apparent false negatives, it would imply approximately 150 true AI misses, with the balance reflecting inconsistencies in the historical benchmark rather than actual AI failures.

After applying the results of that validation sample, estimated recall increased from 75.11% to 98.59%, while the estimated miss rate fell from 24.89% to approximately 1.41%. Overall accuracy increased to 92.52%, and the F1 score increased from 80.35% to 92.19%.

It is equally important not to overstate what the exercise established. The false-positive population was not independently validated in the same manner, so the reported precision figure of 86.38% was left unchanged and could itself be affected by inconsistencies in the historical review.

Human Coding Is a Benchmark, Not an Absolute Truth

The lesson is not that AI is always more accurate than human review, nor does this exercise establish that every disagreement should be resolved in favor of the technology. What it does demonstrate is that historical human coding should not automatically be treated as unquestionable ground truth when evaluating an AI-assisted review.

Human review inevitably involves judgment. Reviewers interpret instructions differently, responsiveness criteria can evolve over the course of a matter, and difficult documents may reasonably generate different conclusions depending on context.

Even well-managed review teams use calibration, quality control and second-level review precisely because some degree of inconsistency is expected. That reality should be taken into account when an AI system is being scored against decisions made during an earlier review.

If every disagreement is automatically attributed to the technology, the resulting metrics may say as much about the consistency of the original review as they do about the AI being tested. A more useful approach is to investigate the disagreement population itself and determine which result is supported by the review criteria.

For legal teams evaluating AI-assisted review, that leads to a straightforward set of questions: What was used as the benchmark, how were disagreements validated, and was the benchmark itself tested for consistency?

A More Conservative Test Than Our Normal Workflow

There was another important limitation to this proof of concept that is worth noting. The exercise was designed specifically to compare aiR Review against an existing review population, so it did not incorporate the full set of AmplifyAI™ analytics and data-reduction techniques Lineal would ordinarily apply before generative AI review in a live matter.

In a typical AmplifyAI workflow, Lineal applies intelligence earlier in the process to reduce irrelevant and low-value material, categorize the population, identify potentially relevant information sooner, and focus subsequent review on the documents most likely to require additional analysis. The broader methodology is designed to reduce review noise before generative AI or human reviewers are asked to make document-level decisions.

That distinction matters because this proof of concept was not intended to demonstrate the maximum amount of data reduction available through the full AmplifyAI methodology. Instead, it was intentionally structured as a relatively direct comparison between AI classifications and the results of a previously completed third-party review.

In a live engagement, aiR Review would ordinarily operate as one component of a broader review methodology rather than as a standalone step applied to an undifferentiated population. Additional analytics, categorization and reduction techniques would typically be applied upstream before determining which documents should be analyzed through generative AI and which should ultimately be promoted for human review.

The results of this exercise were therefore achieved without the benefit of several data-reduction steps that would normally be used to narrow the population earlier in the process. That makes the exercise useful as a validation test because the result was not dependent on the full range of optimization available within the broader AmplifyAI workflow.

Changing the Role of Human Review

None of this suggests that human reviewers are no longer necessary. The more meaningful implication is that AI and analytics may allow legal teams to use human expertise more selectively and in areas where judgment provides the greatest value.

Traditional review generally requires human reviewers to examine the promoted population document by document. An analytics-first workflow can reduce and prioritize that population before review begins, while a more AI-centric model can use generative AI to perform additional categorization and tagging before directing human attention to exceptions, validation and higher-judgment decisions.

That changes the nature of the work rather than simply removing people from the process. Experienced reviewers, subject-matter experts and counsel can spend more time on ambiguous documents, privilege, difficult factual questions, validation and quality control, while technology handles a greater portion of repetitive relevance analysis.

The more productive way to think about AI-assisted review may therefore be less about replacing human judgment and more about concentrating it. Human involvement remains essential, but it does not necessarily need to be applied uniformly across every document in a population.

The Economic Impact Follows the Workflow

The economic implications can be substantial when fewer documents require traditional eyes-on review. In the proof of concept, Lineal modeled three approaches against a population of approximately 29,000 documents to illustrate the potential difference between traditional review, an analytics-assisted AmplifyAI model, and a more AI-driven approach.

Traditional human review was estimated at roughly $60,000, or about $2 per document. The modeled AmplifyAI approach was approximately $19,000, while the AI-assisted model was approximately $15,000, representing potential savings of roughly 69% and 75%, respectively, compared with reviewing the entire population traditionally.

Those figures are illustrative rather than a universal prediction of savings. Document populations, responsiveness rates, review protocols, privilege requirements and matter risk will vary, and each of those factors can affect the appropriate review methodology and resulting economics.

The broader point is that meaningful cost reduction does not come simply from substituting AI for reviewers. It comes from designing a workflow in which analytics, generative AI and targeted human review are applied in the right sequence and to the portions of the population where each is most effective.

What We Learned

Legal teams should continue to scrutinize generative AI carefully. Recall, precision, elusion and other performance metrics matter, and any workflow that reduces traditional human review should include appropriate validation, quality control and documentation.

At the same time, organizations should be careful about how those metrics are established. A historical human review may be the most practical benchmark available, but that does not mean every prior coding decision is necessarily correct.

That was ultimately one of the most useful findings from this exercise. The AI performed well, but the validation process also exposed inconsistencies that already existed in the underlying third-party review, and those inconsistencies materially affected the initial assessment of the technology’s performance.

As AI-assisted review becomes more common, that lesson will become increasingly important. When people and technology disagree, the objective should not be to presume that one or the other must be right, but to use the disagreement as an opportunity to test the result, understand why it occurred, and build a more reliable review process around what we learn.

If you’re evaluating AI-assisted review and want to talk through how to validate it on your own matters, our team can help.

__

About Author

Marco Nasca is the Executive Vice President of Sales at Lineal and a 2001 graduate of DePaul University College of Law. For more than two decades, he has worked at the forefront of eDiscovery and legal technology, advising corporations and law firms on the defensible and practical application of technology in complex litigation, investigations, and regulatory matters. His work focuses on structured data, emerging technologies, artificial intelligence, and the practical implications of evolving judicial doctrine.

__

About Lineal 

Lineal is an innovative eDiscovery and legal technology solutions company that empowers law firms and corporations with modern data management and review strategies. Established in 2009, Lineal specializes in comprehensive eDiscovery services, leveraging its proprietary technology suite, AmplifyAI™ to enhance efficiency and accuracy in handling large volumes of electronic data. With a global presence and a team of experienced professionals, Lineal is dedicated to delivering custom-tailored solutions that drive optimal legal outcomes for its clients. For more information, visit lineal.com