AI Procurement
Essential Considerations in AI Contracting
Procuring AI systems is different. It starts with defining acceptable (and unacceptable) uses of the system and traverses a path that specifies expectations toward data quality, data rights, system quality, transparency, explainability, system monitoring, ongoing risk management, and highly-specialized technical audits and inspections.
Evaluating and controlling these factors are essential in the AI procurement and contracting processes. This is particularly important for systems designed to address needs attached to human rights, human dignity, and civil liberties.
Our responsibility, as procuring officials of such high-risk systems, is to instantiate the values and expectations that our fellow humans deserve. This is no small task, and the playbooks are not well formed.
This playbook, based largely on the work conducted by the City of Amsterdam, provides the procurement community with a helpful set of contracting considerations when evaluating and procuring any high-risk AI system. We hope you will share this widely with colleagues...to achieve a world in which all high-risk AI systems are held to reasonably high standards of responsible design, development, and deployment.
Critical Definitions
Algorithmic System
An "Algorithmic System" is software that:
automatically makes predictions, makes decisions, and/or gives advice
by using data analysis, statistics and/or self-learning logic
CLARIFICATIONS
An Algorithmic System does not require any form of self-learning logic (such as machine learning).
Data analysis may include the combining, cleaning, sorting, classifying, and deriving of data.
Intended Use
“Intended Use” means:
solving the defined problem(s) prior to using the Algorithmic System
used to achieve certain goals but will not, in itself, be able to achieve it
CLARIFICATIONS
“Use” should not only define intended use(s), but also prohibited uses and foreseeable misuses, disuses, and abuses.*
(*requires routine/recurring/continuously available training for administrative users and end-users)
Decisions
“Decisions” are:
administrative, private-law, and/or factual nature, and
that directly or indirectly affect citizens, users, or visitors to a significant extent
CLARIFICATIONS
“Decisions” are broadly interpreted – not a specific decree or decision in the legal meaning of the word.
Factual nature (i.e., municipal trash collection)
“Significant extent” means that it violates any fundamental rights, has legal consequences, or financially impacts a citizen, user, or visitor.
Unacceptable and High Risk Systems
Unacceptable
Systems
Subliminal techniques to distort behavior
Manipulative or deceptive techniques to distort behavior
Exploiting vulnerabilities of individuals or specific groups
Social scoring or evaluating trustworthiness
Risk assessments predicting criminal or administrative offenses
Creating or expanding facial recognition databases through untargeted scraping
Biometric categorization systems based on sensitive attributes or characteristics
Inferring emotions in:
Law enforcement
Border management
The workplace
Education
High-Risk
Systems
educational or vocational training, that may determine the access to education and professional course of someone’s life (e.g. scoring of exams);
employment, management of workers and access to self-employment (e.g. CV-sorting software for recruitment procedures);
financial services (e.g. denying citizens opportunity to obtain a loan);
critical infrastructures and utilities (e.g. electricity, heat, water, Internet or telecommunications access or transportation);
family planning services, including, but not limited to, adoption services or reproductive services,
health care, including, but not limited to, mental health care, dental care or vision care;
housing or lodging, including, but not limited to, any rental, short-term housing or lodging;
law enforcement that may interfere with people’s fundamental rights (e.g. evaluation of the reliability of evidence);
migration, asylum and border control management (e.g. verification of authenticity of travel documents);
administration of justice and democratic processes (e.g. applying the law to a concrete set of facts);
government benefits;
public services;
Remote biometric identification systems;
safety components of products (e.g. AI application in robot-assisted surgery).
Essential Contract Clauses
The e-book contains full descriptions, details, and suggested language for each clause.
Applicability
Data Quality
Data Rights
System Quality
Transparency
Risk Management
System Management
Audits and Inspections
3 Layers of Trust and Understanding
Explainability: Key factors that led an Algorithmic System to a particular result and can be changed to arrive at a different result. [Provided by the developer for user visibility.]
Procedural Transparency: Key factors to understand the processes, methods, and choices used in the development and deployment of the Algorithmic System. [Provided by the developer for user visibility.]
Technical Transparency: Key factors to understand the technical quality and the technical operation of the Algorithmic System. [Evaluated in confidence by a skilled AI auditor.]
See more details and context in the eBook. [Download here.]
About the Authors
Dr. Cari Miller
Founder, Principal, and Lead Researcher at The Center for Inclusive Change. She is recognized globally as one of 100 brilliant women in AI ethics and serves as the Vice Chair of the IEEE Working Group P3119, drafting an international consensus-based standard for AI procurement. She is also a doctoral candidate at Wilmington University, conducting research in AI governance and ethics.
Gisele Waters, PhD.
Founder and Lead researcher at Engineering Hearts® exploring the nature of building human-centered artificial intelligence. She is recognized globally as one of 100 brilliant women in AI ethics and serves as Chair of the IEEE Working Group P3119, drafting an international consensus-based standard for AI procurement. She is also a human-centered design researcher and healthcare service developer. She also advises digital health start-ups on how to build human-centered data science using AI-enabled analytics and remote patient monitoring.
Download a copy of
AI Procurement: Essential Considerations in Contracting