At a glance. Most public-sector rules for artificial intelligence (AI) concentrate on approval before launch, but public value is decided in operation. A credible evaluation tracks six measures: outcomes against a baseline, error rates, equity, trust, recourse and cost. Canada, Ontario, the United States, the United Kingdom (UK) and the European Union (EU) each require parts of this set, and delivery teams can assemble the whole of it.
Approval is not evaluation
Pre-deployment controls are now common. Canada requires an Algorithmic Impact Assessment (AIA) before an automated decision system enters production [1]. United States federal agencies must complete an AI impact assessment before deploying any high-impact AI use [2]. The EU Artificial Intelligence Act (AI Act) requires public bodies to complete a fundamental rights impact assessment before the first use of certain high-risk systems [3].
These instruments also look past launch. Canada’s Directive on Automated Decision-Making requires departments to monitor outcomes on a scheduled basis and to publish information on each system’s effectiveness and efficiency in meeting program objectives [1]. United States agencies must monitor high-impact AI for performance and adverse impacts, and that monitoring must be designed to detect changes after deployment [2]. The AI Principles of the Organisation for Economic Co-operation and Development (OECD), updated in 2024, list accountability among their five values-based principles [4].
What most instruments leave to departments is the specific measurement set. That is where delivery teams can add the most value.
Six measures of public value
| Measure | Question it answers | Example indicator |
|---|---|---|
| Outcomes | Is the service better than before? | Processing time against a pre-AI baseline |
| Error rates | How often is the system wrong, and how? | False positives and false negatives by decision type |
| Equity | Does performance differ across groups? | Error rates by group, where data allows |
| Trust | Do clients and staff rely on it appropriately? | Client satisfaction; staff override rate |
| Recourse | Can people challenge results effectively? | Appeals, reversals, time to human review |
| Cost | Is the value worth the full run cost? | Cost per case, including oversight |
1. Outcomes against a baseline
An outcome means something only against a baseline. United States guidance is explicit. The AI impact assessment must state the intended purpose and expected benefit, supported by specific metrics or qualitative analysis, compared with existing agency processes [2]. Acquisition guidance adds a periodic evaluation of a system’s value to the government that weighs effectiveness, risk and ongoing costs [5]. In Canada, the AI Strategy for the Federal Public Service 2025-2027 pairs transparency with value to Canadians in one of its four priority areas [6].
2. Error rates by type
A single accuracy figure hides what matters in public services. A false negative in an eligibility check can deny a benefit. A false positive in a fraud screen can start an investigation of someone who did nothing wrong. Track each error type separately, for each decision type.
The UK’s AI Playbook draws a related line between model metrics, which show technical performance, and service metrics, which show whether user needs and business goals are met. It notes that a highly accurate model can still fail if users do not trust it [7].
3. Equity across groups
Canada requires departments to test data and models for unintended biases before production, and to complete a Gender-based Analysis Plus (GBA Plus) for systems at impact Level II and above [1]. United States impact assessments must describe information in the data about classes protected by federal non-discrimination laws, where applicable [2]. The EU assessment must identify the groups of people likely to be affected and the specific risks of harm to them [3].
In operation, the test is practical. Are error rates and outcomes materially different across groups, and is someone accountable for looking?
4. Trust, measured in behaviour
Trust shows up in behaviour as well as in surveys. Two indicators help. Client satisfaction shows whether people accept the service. The staff override rate shows how officers use the system. A very high rate may signal a weak model. A very low rate on complex cases may signal over-reliance. The UK Playbook recommends user research on trust, and feedback channels that let users report issues and trigger human review [7].
5. Recourse that works
Canada, the United States, Ontario and the EU each provide some route to challenge decisions or ask questions. Canada requires departments to give clients recourse options to challenge the decision [1]. United States agencies must give people affected by AI-enabled decisions access to timely human review and a chance to appeal, where appropriate [2]. Ontario requires ministries to provide an accessible way to seek information about AI use, while stating that this does not create a new appeal route [8]. The EU assessment must set out internal governance and complaint arrangements [3].
The measure is not whether a route exists. It is how many people use it, how often decisions are reversed, and how long a human review takes.
6. Full cost
AI services carry run costs that pilots hide: monitoring, retraining, human review and vendor fees. United States guidance asks agencies to weigh ongoing operation and maintenance costs in periodic value reviews [5]. A cost per case that includes oversight is a fairer comparison with the old process than licence cost alone.
How jurisdictions compare
| Jurisdiction | In operation | Public reporting |
|---|---|---|
| Canada (federal) | Scheduled outcome monitoring | AIA results; effectiveness information; AI Register |
| Ontario | Risk monitoring; annual reporting | Published list of AI use cases |
| United States | Ongoing monitoring; appeals | Annual use case inventories |
| United Kingdom | Testing and monitoring across the life cycle | Algorithmic transparency records |
| European Union | Rights impact assessment kept current | Results notified to the market surveillance authority |
Canada’s federal AI Register, first published in November 2025, lists more than 400 systems across 42 institutions [9]. Ontario’s directive requires ministries to report annually on AI use cases and risk management, and to publish a list of AI use cases [8]. United States agencies had until April 2026 to document minimum practices for high-impact AI, and must stop using high-impact AI that does not comply [2]. UK central government had published 59 records under its Algorithmic Transparency Recording Standard by May 2025 [10]. Under the EU AI Act, a deployer must update its assessment when relevant elements change and notify the market surveillance authority of the results [3].
The pattern is consistent. Most public reporting today describes which systems exist and how they were assessed. Routine public reporting of operating results, such as error rates by type or the share of appealed decisions that are reversed, is less common. Departments can take that step without waiting for a mandate.
Before go-live: five agreements
My work is the delivery of manufacturing intelligence, industrial Internet of Things (IIoT) and manufacturing execution system (MES) programs, where measurable outcomes are the agreement with the business. The lesson that transfers is simple: a dashboard is not a performance system until each number has an owner and a threshold that triggers action. Five agreements put that into practice.
- Capture the baseline. Measure the current process before the AI service replaces or supports it.
- Name an owner for each measure. One accountable person per indicator, not one committee for all of them.
- Set thresholds that trigger action. Define the point that triggers review and the point that suspends use.
- Plan equity data lawfully. Agree with privacy specialists early what group-level data can be collected and how.
- Fix the reporting rhythm. For example, monthly to the service owner, quarterly to executives and annually to the public.
The bottom line
An AI service earns its place through results in operation, not approval at launch. Six measures, a baseline and named owners give leaders an honest view of public value. Publishing them gives the public a reason to trust it.
References
- Treasury Board of Canada Secretariat, “Directive on Automated Decision-Making,” Government of Canada, 2019, as amended.
- Office of Management and Budget, “Accelerating Federal Use of AI through Innovation, Governance, and Public Trust,” Memorandum M-25-21, Executive Office of the President, Washington, DC, USA, Apr. 3, 2025.
- European Parliament and Council of the European Union, “Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act),” Official Journal of the European Union, Jul. 12, 2024, Art. 27.
- OECD, “OECD AI Principles” (Recommendation of the Council on Artificial Intelligence, OECD/LEGAL/0449), adopted May 2019, updated May 2024.
- Office of Management and Budget, “Driving Efficient Acquisition of Artificial Intelligence in Government,” Memorandum M-25-22, Executive Office of the President, Washington, DC, USA, Apr. 3, 2025.
- Treasury Board of Canada Secretariat, “AI Strategy for the Federal Public Service 2025-2027,” Government of Canada, Mar. 2025.
- Government Digital Service, “Artificial Intelligence Playbook for the UK Government,” GOV.UK, Feb. 10, 2025.
- Government of Ontario, “Responsible Use of Artificial Intelligence Directive,” Management Board of Cabinet, effective Dec. 1, 2024, updated Jan. 2026.
- Treasury Board of Canada Secretariat, “Canada launches first register of AI uses in federal government,” news release, Ottawa, ON, Canada, Nov. 28, 2025.
- E. Hess-Rheingans, A. Hart, S. Handyside and S. Beaugrand, “Making the Algorithmic Transparency Recording Standard (ATRS) mandatory across government,” Data in Government blog, Government Digital Service, May 8, 2025.