AI Procurement Risk Management Framework
Risk Management Framework for AI Procurement
Classifying and understanding risk is essential when procuring an AI system. It is important to recognize that the types and scale of risks vary from system to system.
When AI systems are developed for high-risk domains (e.g., employment, health, education, housing, finance, public assistance, etc.), two risk indicators become highly relevant. These indicators include:
1) the complexities within the AI system, and
2) the impact that outcome(s) may have on human lives.
Hence, it is imperative to determine how much risk the procuring organization is willing to accept for each system at the outset of each procurement. This act is known as establishing the risk appetite for the procurement. A well-defined risk appetite for a procurement should serve as an anchoring point throughout the procurement lifecycle to guide risk mitigation strategies and create an acceptable risk tolerance for the chosen system.
RMF Overview
Unacceptable and High Risk Systems
The RMF for AI Procurement becomes increasingly more important when high-risk systems are at hand.
Unacceptable
Systems
Subliminal techniques to distort behavior
Manipulative or deceptive techniques to distort behavior
Exploiting vulnerabilities of individuals or specific groups
Social scoring or evaluating trustworthiness
Emotional recognition in the workplace and education settings
Risk assessments predicting criminal or administrative offenses
Creating or expanding facial recognition databases through untargeted scraping
Biometric categorization systems based on sensitive attributes or characteristics
Inferring emotions in:
Law enforcement
Border management
The workplace
Education
High-Risk
Systems
educational or vocational training, that may determine the access to education and professional course of someone’s life (e.g. scoring of exams);
employment, management of workers and access to self-employment (e.g. CV-sorting software for recruitment procedures);
financial services (e.g. denying citizens opportunity to obtain a loan);
critical infrastructures and utilities (e.g. electricity, heat, water, Internet or telecommunications access or transportation);
family planning services, including, but not limited to, adoption services or reproductive services,
health care, including, but not limited to, mental health care, dental care or vision care;
housing or lodging, including, but not limited to, any rental, short-term housing or lodging;
law enforcement that may interfere with people’s fundamental rights (e.g. evaluation of the reliability of evidence);
migration, asylum and border control management (e.g. verification of authenticity of travel documents);
administration of justice and democratic processes (e.g. applying the law to a concrete set of facts);
government benefits;
public services;
Remote biometric identification systems;
safety components of products (e.g. AI application in robot-assisted surgery).
Steps to Establishing a Risk Appetite
Every AI use case is unique. Hence, every procurement will require a risk appetite that guides the team through the process of assessing and controlling relevant risks. The paper provides a scorecard, risk appetite matrix, convenient risk appetite statements, and suggested risk mitigation alignments to support a the responsible procurement of AI systems.
About the Authors
Dr. Cari Miller
Founder and Lead Researcher at The Center for Inclusive Change and Co-Founder/Board Chair of the AI Procurement Lab. Cari is a renowned AI risk expert, AI governance researcher, and consultant. She advises organizations on AI risk management, governance considerations, and responsible AI procurement. She is a certified change manager and follows an inclusive organizational design philosophy.
Gisele Waters, PhD.
Co-Founder and CEO of the AI Procurement Lab, AI governance and procurement standards builder, human-centered service designer, researcher, and culturally responsive evaluator. Gisele has built multidisciplinary guidance and tools over 25 years in education, healthcare, and information technology all threaded together by her passion for mitigating risk to vulnerable populations and communities.
Download a copy of the
Risk Management Framework for the Procurement of AI Systems