Buying AI Responsibly: What Governments Expect from AI Suppliers

At a glance. Governments now govern much of their artificial intelligence (AI) through the procurement contract. Across Canada, the United States, the United Kingdom (UK) and the European Union (EU), buyers are asking for the same evidence: tested performance, documentation for impact assessments, limits on data use, audit access and a credible exit plan. Suppliers that prepare it early make procurement faster, and buyers that write it into requirements reduce delivery risk.

Why procurement is where AI governance becomes enforceable

A policy states what a public body must do. A contract states what a supplier must do so that the public body can comply. When a department buys an AI system, its duties to assess impact, explain decisions and monitor outcomes depend on information and access that the supplier holds. If the contract is silent, those duties are hard to meet.

That is why recent public guidance treats procurement as a control point rather than an administrative step.

Canada

The federal Directive on Automated Decision-Making applies to automated decision systems developed or procured after April 1, 2020 [1]. Procured systems carry two obligations that suppliers should expect to see reflected in contracts.

First, where a proprietary licence is used, the Government of Canada keeps the right to access and test the system, including released versions of proprietary components, when needed for an audit, investigation, inspection, examination, enforcement action or judicial proceeding. It may also authorize external parties to review and audit those components [1].

Second, the department must complete and release an Algorithmic Impact Assessment (AIA) before production and, from impact Level II upward, arrange a peer review [1]. Both depend on supplier documentation.

For sourcing, Public Services and Procurement Canada (PSPC) and the Treasury Board of Canada Secretariat (TBS) ran a process that produced a list of pre-qualified AI suppliers, known as the AI Source List. Departments can use it to start a streamlined procurement [2].

United States

Memorandum M-25-22 from the Office of Management and Budget (OMB), issued on April 3, 2025, sets federal rules for acquiring AI. It replaces M-24-18 and applies to contracts awarded under solicitations issued 180 days or more after issuance, which is the end of September 2025 [3]. Its main expectations include:

• Data-use limits. Agency processes should ensure that contracts permanently prohibit using non-public agency data, and the results produced from it, to further train publicly or commercially available AI without explicit agency consent [3].
• Lock-in protections. Contracts must address knowledge transfer, data and model portability, rights to code and models produced under the contract, and transparency in licensing and pricing [3].
• Buyer testing. Agencies must, as far as practicable, test proposed solutions during evaluation. Contracts must let the agency monitor performance regularly, for example quarterly or twice a year, using agency-defined data the vendor cannot access [3].
• Impact disclosure. Where practicable, solicitations must state whether a use is likely to be high-impact, and vendors should supply the information the agency needs for its AI impact assessment [3]. High-impact AI is defined in the companion memorandum, M-25-21 [4].

United Kingdom

The Guidelines for AI procurement, published in June 2020 by the Office for Artificial Intelligence with the World Economic Forum and UK government commercial bodies, set out ten considerations [5]. Three stand out for suppliers: buyers should assess their data before going to market, focus on the challenge rather than a specific solution, and avoid black-box algorithms and vendor lock-in. The guidelines also ask buyers to plan the system’s whole life cycle, including end of life [5].

The AI Playbook for the UK Government, published by the Government Digital Service (GDS) in February 2025, adds a principle that teams work with commercial colleagues from the start. It says requirements should address data strategy, bias mitigation, vendor lock-in, intellectual property and liability. It also notes that contracts can require suppliers to provide information in the categories of the Algorithmic Transparency Recording Standard (ATRS) [6].

The ATRS scope policy asks public bodies to set transparency expectations with suppliers early, including in tender documents, and to apply commercial confidentiality exemptions narrowly [7]. By May 2025, 59 ATRS records had been published [8].

European Union

The EU Artificial Intelligence Act (AI Act) places most obligations for high-risk systems on providers. The main high-risk rules apply from August 2, 2026, and obligations for general-purpose AI models have applied since August 2, 2025 [9]. Public bodies that deploy certain high-risk systems must complete a fundamental rights impact assessment, drawing on the information the provider supplies with the system [9].

To help buyers, the European Commission’s Public Buyers Community published model contractual AI clauses in 2023, in high-risk and non-high-risk versions. They are voluntary, are designed to be attached as a schedule to a contract, and do not cover intellectual property, acceptance or payment [10].

What the jurisdictions share

The legal force differs, from binding regulation in the EU to policy and guidance elsewhere. The expectations converge. They also reflect the transparency, robustness and accountability principles in the AI Principles of the Organisation for Economic Co-operation and Development (OECD), which Canada, the United States and the United Kingdom have adopted [11].

Seven things suppliers should prepare

1. Performance evidence on the buyer’s terms. Expect testing on data that resembles real operating conditions, sometimes on data you cannot see.
2. An impact assessment package. Intended use, training and input data, known limits and recommended human oversight, written so a public servant can use it in an AIA or its equivalent.
3. Audit and access arrangements. Decide in advance how the buyer or an external reviewer can test the system, including proprietary components.
4. Clear data-use terms. State what happens to the buyer’s data and outputs, and confirm they will not train other models without consent.
5. A portability and exit plan. Formats, interfaces, knowledge transfer and handover steps at contract end.
6. Publishable descriptions. Plain-language text a public body can use in a transparency record or a published assessment.
7. Change control. Notice before new AI features are added, and the ability to roll back a version that fails agreed standards. M-25-22 encourages agencies to require both [3].

A note for buyers

I work in industrial technology delivery, across industrial Internet of Things (IIoT), manufacturing execution system (MES) and manufacturing intelligence programs. The lesson that transfers most directly is about acceptance. Acceptance criteria written before award are enforced. Acceptance criteria written after award are negotiated.

For AI, that means defining the test data, performance thresholds, monitoring rights and exit terms in the requirements, not at the first steering committee after award.

The bottom line

Responsible AI procurement is not separate from responsible AI. It is the point where a public body’s duties become a supplier’s deliverables. Buyers and suppliers who treat that list as shared scope, rather than as friction, will spend less time negotiating and more time delivering.

References

1. Treasury Board of Canada Secretariat, “Directive on Automated Decision-Making,” Government of Canada, 2019, as amended.
2. Treasury Board of Canada Secretariat and Public Services and Procurement Canada, “List of interested Artificial Intelligence (AI) suppliers” (AI Source List, solicitation EN578-180001/B), Government of Canada.
3. Office of Management and Budget, “Driving Efficient Acquisition of Artificial Intelligence in Government,” Memorandum M-25-22, Executive Office of the President, Washington, DC, USA, Apr. 3, 2025.
4. Office of Management and Budget, “Accelerating Federal Use of AI through Innovation, Governance, and Public Trust,” Memorandum M-25-21, Executive Office of the President, Washington, DC, USA, Apr. 3, 2025.
5. Office for Artificial Intelligence, “Guidelines for AI procurement,” GOV.UK, Jun. 8, 2020.
6. Government Digital Service, “Artificial Intelligence Playbook for the UK Government,” GOV.UK, Feb. 10, 2025.
7. Government Digital Service, “Algorithmic Transparency Recording Standard (ATRS) Mandatory Scope and Exemptions Policy,” GOV.UK, Dec. 17, 2024.
8. E. Hess-Rheingans, A. Hart, S. Handyside and S. Beaugrand, “Making the Algorithmic Transparency Recording Standard (ATRS) mandatory across government,” Data in Government blog, Government Digital Service, May 8, 2025.
9. European Parliament and Council of the European Union, “Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act),” Official Journal of the European Union, Jul. 12, 2024.
10. European Commission, Public Buyers Community, “EU model contractual AI clauses to pilot in procurements of AI,” Sep. 29, 2023.
11. OECD, “OECD AI Principles” (Recommendation of the Council on Artificial Intelligence, OECD/LEGAL/0449), adopted May 2019, updated May 2024.