Measure Before You Trust: Evaluating AI Systems Like a Scientific Instrument

Bottom line. A demonstration shows that an artificial intelligence (AI) system can produce a good answer. It does not show how often it does. Before an AI system supports real decisions, measure it the way a laboratory measures an instrument: against a fixed test set, with a stated error rate, a confidence interval and known failure modes. Claims the system cannot support with a source should fail rather than ship.

A demonstration is a sample of one

Language models are probabilistic. The same question can produce different answers on different runs, and a small change in wording can change the result. A demonstration typically shows a handful of well-chosen examples. It tells you the system is capable of a good answer, not how often it gives one.

The cost of that gap is now on the public record. In Moffatt v. Air Canada (2024), a British Columbia tribunal held the airline responsible for incorrect information about bereavement fares given by the chatbot on its website, and rejected the argument that the chatbot was responsible for its own statements [1]. In Mata v. Avianca (2023), a United States federal court sanctioned lawyers who filed a brief citing court decisions that a chatbot had invented [2]. In both cases the system produced fluent, confident text that no one had checked against a source.

Treat the system as an instrument

A scientific instrument is trusted because its behaviour is characterized: its accuracy, its precision, the conditions under which it is valid and how it fails. An AI system deserves the same treatment. In practice that means four things:

1. A fixed test set drawn from real use. Questions and documents that represent the actual task, with answers agreed by people who know the domain. Public benchmarks are a starting point, but a model may have seen them during training. A high score then measures memory rather than capability.
2. A stated error rate with its uncertainty. Not “it works well”, but “it answered 91% of 400 test questions correctly, plus or minus 3 points”.
3. Known failure modes. Which kinds of question it gets wrong, and whether it fails loudly (refuses or flags uncertainty) or quietly (answers confidently and incorrectly).
4. Repeat measurement. Every change to the model, the prompt or the source documents is a change to the instrument. Re-run the test set each time.

How large does the test set need to be?

Small test sets are the most common weakness in AI evaluation, and the arithmetic is simple. Suppose a system is correct 90% of the time. The margin of error on its measured accuracy depends on how many test cases are used:

These values use the standard normal approximation to the binomial distribution. The practical rule follows directly. On 100 test cases, a 3-point difference between two models or two prompts is not evidence of a difference. Choosing which model to deploy needs either a larger test set or a larger difference.

Citations are a test, not a decoration

For systems that answer questions from documents, the most useful single control is to require every factual claim to cite a source passage, and to check that the passage supports the claim. If the system cannot cite a source, the claim should fail rather than be shown with a warning. This turns an invisible failure (a fluent, invented answer) into a visible one (no answer), which is far cheaper to handle.

Citation checking also makes the system auditable. A reviewer can verify a cited answer in seconds. An uncited answer has to be researched from scratch, which removes most of the time the system was meant to save.

Use the frameworks that already exist

Organizations do not need to invent an evaluation discipline. Two widely used frameworks provide the structure:

• NIST AI Risk Management Framework (AI RMF 1.0). Published by the United States National Institute of Standards and Technology (NIST) in January 2023, it organizes AI risk work into four functions: Govern, Map, Measure and Manage [3]. NIST’s Generative AI Profile, released in July 2024, applies the framework to systems built on large language models [4].
• ISO/IEC 42001:2023. Published in December 2023 by the International Organization for Standardization and the International Electrotechnical Commission, it sets requirements for establishing, operating and continually improving an AI management system [5].

Neither framework says what accuracy is good enough. That depends on the decision the system supports and the cost of a wrong answer. Both require the question to be asked, and answered with evidence.

How we apply it

Our rule is that an AI capability moves from prototype to product only with a written evaluation: the test set, the measured error rate with its interval, the known failure modes and the person accountable for acting on the output. The evaluation is re-run whenever the model, the prompt or the source documents change. This adds days to a project. It removes the much larger cost of discovering the error rate in production.

Conclusion

AI systems are now capable enough that the limiting factor is rarely what they can do once. It is what they do reliably, and whether anyone can tell the difference. Measuring them like instruments, with stated error rates and enforced citations, is how a promising demonstration becomes a system an organization can depend on.

References

1. Moffatt v. Air Canada, 2024 BCCRT 149, Civil Resolution Tribunal of British Columbia, Feb. 2024.
2. Mata v. Avianca, Inc., No. 22-cv-1461 (PKC), U.S. District Court for the Southern District of New York, Jun. 2023.
3. National Institute of Standards and Technology, “Artificial Intelligence Risk Management Framework (AI RMF 1.0),” NIST AI 100-1, Jan. 2023.
4. National Institute of Standards and Technology, “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile,” NIST AI 600-1, Jul. 2024.
5. International Organization for Standardization, “ISO/IEC 42001:2023 Information technology: Artificial intelligence: Management system,” Dec. 2023.