AI Explainability

AI Explainability in Contracting

Explainability instills trust. When done well, the magic of explainable AI (XAI) will lead to a trusted system. This means providing an understanding of the system by being transparent about the system itself and how it was developed with respect to decisions towards fairness, reliability, and accountability.

Explainability isn't just for end users. This is a very narrow definition of explainability that we've held to for too long. Yes, it is an important narrow definition. However, when we look at the broader landscape of an AI system, there are many more stakeholders that require explainability.


This reference guide broadens the view of explainability and offers critical insights into key XAI considerations during procurement and when contracting with AI providers. 

Key Stakeholders

AI explainability runs on a continuum - from highly technical explainability needs to clear and plain language needs. 

Each stakeholder has unique explainability needs.

Auditors & Assessors

Integrators & Security Staff

Procurement Professionals

Administrative Users

Impacted End Users

What matters most when it comes to explainability?

Models Matter

Simply put, some models are more transparent and explainable than others. Model choice matters.

Context Matters

Explainability is a matter of human dignity when the the output results in a life-impacting decision.

Words Matter

AI systems are complex and highly technical. Only some stakeholders speak geek.

Timing Matters

Some stakeholders need the info all at once, others need it sprinkled across the entire system. 

When it Matters Most

Explainability is a fundamental matter of human dignity when it comes to high-risk systems operated in the following domains:

Health

Employment

Education

Housing

Financial Services

Public Services / Benefits

Utilities / Infrastructure

Law Enforcement

Justice & Legal Services

Immigration

Biometrics & Geo-Tracking

Product Safety Components

About the Authors

Dr. Cari L. Miller

Founder, Principal, and Lead Researcher at The Center for Inclusive Change. She is recognized globally as one of 100 brilliant women in AI ethics and serves on the board of ForHumanity, an international non-profit organization developing AI audit criteria. She is also the Vice Chair of the IEEE Working Group P3119 drafting an international consensus-based standard for AI procurement. She holds a doctorate degree from Wilmington University with a research focus in AI governance and ethics.

Dr. Gisele Waters

Co-Founder and CEO of the AI Procurement Lab, AI governance and procurement standards builder, human-centered service designer, researcher, and culturally responsive evaluator. Gisele has built multidisciplinary guidance and tools over 25 years in education, healthcare, and information technology all threaded together by her passion for mitigating risk to vulnerable populations and communities.  

Download a copy of 

AI Procurement: Explainability Best Practices