AI Procurement

Essential Considerations in AI Contracting

Procuring AI systems is different. It starts with defining acceptable (and unacceptable) uses of the system and traverses a path that specifies expectations toward data quality, data rights, system quality, transparency, explainability, system monitoring, ongoing risk management, and highly-specialized technical audits and inspections.

Evaluating and controlling these factors are essential in the AI procurement and contracting processes. This is particularly important for systems designed to address needs attached to human rights, human dignity, and civil liberties.

Our responsibility, as procuring officials of such high-risk systems, is to instantiate the values and expectations that our fellow humans deserve. This is no small task, and the playbooks are not well formed.

This playbook, based largely on the work conducted by the City of Amsterdam, provides the procurement community with a helpful set of contracting considerations when evaluating and procuring any high-risk AI system. We hope you will share this widely with colleagues...to achieve a world in which all high-risk AI systems are held to reasonably high standards of responsible design, development, and deployment.

Download Now

Critical Definitions

Algorithmic System

An "Algorithmic System" is software that:

automatically makes predictions, makes decisions, and/or gives advice
by using data analysis, statistics and/or self-learning logic

CLARIFICATIONS

An Algorithmic System does not require any form of self-learning logic (such as machine learning).

Data analysis may include the combining, cleaning, sorting, classifying, and deriving of data.

Intended Use

“Intended Use” means:

solving the defined problem(s) prior to using the Algorithmic System
used to achieve certain goals but will not, in itself, be able to achieve it

CLARIFICATIONS

“Use” should not only define intended use(s), but also prohibited uses and foreseeable misuses, disuses, and abuses.*

(*requires routine/recurring/continuously available training for administrative users and end-users)

Decisions

“Decisions” are:

administrative, private-law, and/or factual nature, and
that directly or indirectly affect citizens, users, or visitors to a significant extent

CLARIFICATIONS

“Decisions” are broadly interpreted – not a specific decree or decision in the legal meaning of the word.

Factual nature (i.e., municipal trash collection)

“Significant extent” means that it violates any fundamental rights, has legal consequences, or financially impacts a citizen, user, or visitor.

Unacceptable and High Risk Systems

Unacceptable

Systems

Subliminal techniques to distort behavior
Manipulative or deceptive techniques to distort behavior
Exploiting vulnerabilities of individuals or specific groups
Social scoring or evaluating trustworthiness
Risk assessments predicting criminal or administrative offenses
Creating or expanding facial recognition databases through untargeted scraping
Biometric categorization systems based on sensitive attributes or characteristics
Inferring emotions in:

Law enforcement
Border management
The workplace
Education

High-Risk

Systems

educational or vocational training, that may determine the access to education and professional course of someone’s life (e.g. scoring of exams);

employment, management of workers and access to self-employment (e.g. CV-sorting software for recruitment procedures);

financial services (e.g. denying citizens opportunity to obtain a loan);

critical infrastructures and utilities (e.g. electricity, heat, water, Internet or telecommunications access or transportation);

family planning services, including, but not limited to, adoption services or reproductive services,

health care, including, but not limited to, mental health care, dental care or vision care;

housing or lodging, including, but not limited to, any rental, short-term housing or lodging;

law enforcement that may interfere with people’s fundamental rights (e.g. evaluation of the reliability of evidence);

migration, asylum and border control management (e.g. verification of authenticity of travel documents);

administration of justice and democratic processes (e.g. applying the law to a concrete set of facts);

government benefits;

public services;

Remote biometric identification systems;

safety components of products (e.g. AI application in robot-assisted surgery).

Essential Contract Clauses

The e-book contains full descriptions, details, and suggested language for each clause.

[Download Here]

Applicability

Data Quality

Data Rights

System Quality

Transparency

Risk Management

System Management

Audits and Inspections

3 Layers of Trust and Understanding

Explainability: Key factors that led an Algorithmic System to a particular result and can be changed to arrive at a different result. [Provided by the developer for user visibility.]

Procedural Transparency: Key factors to understand the processes, methods, and choices used in the development and deployment of the Algorithmic System. [Provided by the developer for user visibility.]

Technical Transparency: Key factors to understand the technical quality and the technical operation of the Algorithmic System. [Evaluated in confidence by a skilled AI auditor.]

See more details and context in the eBook. [Download here.]

About the Authors

Dr. Cari Miller

Founder, Principal, and Lead Researcher at The Center for Inclusive Change. She is recognized globally as one of 100 brilliant women in AI ethics and serves as the Vice Chair of the IEEE Working Group P3119, drafting an international consensus-based standard for AI procurement. She is also a co-founder of the AI Procurement Lab and a senior researcher in AI governance and ethics.

Gisele Waters, PhD.

Co-Founder and CEO of the AI Procurement Lab, AI governance and procurement standards builder, human-centered service designer, researcher, and culturally responsive evaluator. Gisele has built multidisciplinary guidance and tools over 25 years in education, healthcare, and information technology all threaded together by her passion for mitigating risk to vulnerable populations and communities..