Abstract
In this paper, we distinguish between different sorts of assessments of algorithmic systems, describe our process of assessing such systems for ethical risk, and share some key challenges and lessons for future algorithm assessments and audits. Given the distinctive nature and function of a third-party audit, and the uncertain and shifting regulatory landscape, we suggest that second-party assessments are currently the primary mechanisms for analyzing the social impacts of systems that incorporate artificial intelligence. We then discuss two kinds of assessments: an ethical risk assessment and a narrower, technical algorithmic bias assessment. We explain how the two assessments depend on each other, highlight the importance of situating the algorithm within its particular socio-technical context, and discuss a number of lessons and challenges for algorithm assessments and, potentially, for algorithm audits. The discussion builds on our team’s experience of advising and conducting ethical risk assessments for clients across different industries in the last 4 years. Our main goal is to reflect on the key factors that are potentially ethically relevant in the use of algorithms and draw lessons for the nascent algorithm assessment and audit industry, in the hope of helping all parties minimize the risk of harm from their use.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability
Data sharing not applicable to this article as no datasets are directly relevant to the content of this article.
Notes
We use “ethical risk assessment” rather than “ethical impact assessment” because the latter naturally suggests the actual impact or consequences of the use of an algorithm, while the former covers all ethical risks, whether they come to be realized or not. We recognize, however, that “impact” is often used in the broader sense that covers risk.
See, for example, the Canadian government’s (2021) algorithmic impact assessment tool: https://www.canada.ca/en/government/system/digital-government/digital-government-innovations/responsible-use-ai/algorithmic-impact-assessment.html
An exception is a case study by Mökander and Floridi (2022). They provide a detailed case study based on an observation of an “ethics-based audit” of AztraZeneca conducted by a third party (not by the authors).
Carrier and Brown’s (2021) taxonomy also includes an “internal audit” which is carried out by a group or unit working independently within and in service of the organization rather than society or some other party but otherwise has the characteristics of an audit.
See, for example, the Ada Lovelace Institute (2020) report: https://www.adalovelaceinstitute.org/report/examining-the-black-box-tools-for-assessing-algorithmic-systems/
In some cases, one might be unfairly treated or discriminated against without being worse off than one otherwise would have been, and at least in this sense one might not be “harmed” by unfairness. We use ‘harm’ in a broader sense that includes such cases of unfair or discriminatory treatment.
For example, see Watcher et al.’s (2021) “conditional demographic disparity,” and IBM Research’s list of metrics on the AI fairness 360 site: https://aif360.mybluemix.net
See, for example, Ben-Shahar and Schneider (2014). Thanks to an anonymous referee for raising this important point.
References
Ada Lovelace Institute. (2020). Examining the Black Box: Tools for assessing algorithmic systems. Retrieved February 20, 2022. https://www.adalovelaceinstitute.org/report/examining-the-black-box-tools-for-assessing-algorithmic-systems/
Andrus, M., & Villeneuve, S. (2022). Demographic-reliant algorithmic fairness: Characterizing the risks of demographic data collection in the pursuit of fairness. In 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’22), June 21–24, 2022, Seoul, Republic of Korea. ACM, New York, NY, USA, 21 pages. https://doi.org/10.1145/3531146.3533226
Basl J., Sandler, R., and Tiel, S. (2021). Getting from commitment to content in AI and data ethics: Justice and explainability. Atlantic Council. Retrieved May 30, 2022. https://www.atlanticcouncil.org/in-depth-research-reports/report/specifying-normative-content/?mkt_tok=NjU5LVdaWC0wNzUAAAF_slunuNBmXLNnheGh0w-KgEPaF8uewmUN3T7b1fFhbKHlDLa-V9Hw7UxOQVcPMrTBbngaUICIzBLDNXD7S30ZcxaKgKSvyTD6BF69Z2MH
Baum, K., Mantel, S., Speith, T., & Schmidt, E. (2022). From responsibility to reason-giving explainable artificial intelligence. Philosophy and Technology, 35(1), 1–30.
Benjamin, R. (2019). Race after technology: Abolitionist tools for the New Jim Code. Polity.
Ben-Shahar, O., & Schneider, C. E. (2014). More than you wanted to know: The failure of mandated disclosure. Princeton University Press.
Brennan-Marquez, K. (2017). Plausible cause: Explanatory standards in the age of powerful machines. Vanderbilt Law Review, Vol. 70.
Brown, S., Davidovic, J., & Hasan, A. (2021). The algorithm audit: Scoring the algorithms that score us. Big Data & Society. https://doi.org/10.1177/2F2053951720983865
Buolamwini, J., & Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. Proceedings of Machine Learning Research, 81, 1–15.
Canadian Government. (2021). Algorithmic impact assessment tool. Retrieved February 10, 2022. https://www.canada.ca/en/government/system/digital-government/digital-government-innovations/responsible-use-ai/algorithmic-impact-assessment.html
Carrier, R., & Brown, S. (2021). Taxonomy: AI audit, assurance, and assessment. Retrieved February 20, 2022. https://forhumanity.center/blog/taxonomy-ai-audit-assurance-assessment/
Dotan, R. (2021). Theory choice, non-epistemic values, and machine learning.Synthese 198, 11081–11101. https://doi.org/10.1007/s11229-020-02773-2
Fazelpour, S., & Danks, D. (2021). Algorithmic bias: Senses, sources, solutions. Philosophy Compass, 16(8), e12760. https://doi.org/10.1111/phc3.12760
IBM Research. AI Fairness 360. Retrieved May 30, 2022. https://aif360.mybluemix.net
Liao, M. (2020). Ethics of artificial intelligence. Oxford University Press.
Matthias, A. (2004). The responsibility gap: Ascribing responsibility for the actions of learning automata. Ethics and Information Technology, 6(3), 175–183.
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, AR. (2022). A survey on bias and fairness in machine learning. Retrieved February 20, 2022, from the arXiv database. https://arxiv.org/abs/1908.09635
Mittelstadt, B. (2019). Explaining explanations in AI. FAT* 2019 Proceedings 1. https://doi.org/10.1145/3287560.3287574
Mökander, J., & Floridi, L. (2021). Ethics-based auditing to develop trustworthy AI. Minds & Machines, 31, 323–327. https://doi.org/10.1007/s11023-021-09557-8
Mökander, J., & Floridi, L. (2022). Operationalising AI governance through ethics-based auditing: an industry case study. AI Ethics. https://doi.org/10.1007/s43681-022-00171-7
Moss, E., Watkins, E.A., Singh, R., Elish, M.C., & Metcalf, J. (2021). Assembling accountability: Algorithmic impact assessment for the public interest. Retrieved February 20, 2020. https://datasociety.net/library/assembling-accountability-algorithmic-impact-assessment-for-the-public-interest/
New York City Council. (2021). A Local Law to amend the administrative code of the city of New York, in relation to automated employment decision tools. Retrieved February 20, 2022. https://legistar.council.nyc.gov/LegislationDetail.aspx?ID=4344524&GUID=B051915D-A9AC-451E-81F8-6596032FA3F9
OECD. AI principles overview. Retrieved May 30, 2022. https://oecd.ai/en/ai-principles
Russell, S., & Norvig, P. (2010). Artificial intelligence: A modern approach (3rd ed.). Prentice Hall.
Sandler, R., & Basl, J. (2019). Building Data and AI Ethics Committees. https://www.accenture.com/_acnmedia/PDF-107/Accenture-AI-And-Data-Ethics-Committee-Report-11.pdf#zoom=50
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019). Fairness and abstraction in sociotechnical systems. FAT* ‘19: Proceedings of the Conference on Fairness, Accountability, and Transparency, 59–69. https://doi.org/10.1145/3287560.3287598
Selbst, A. D. (2021). An institutional view of algorithmic impact assessments. 35 Harvard Journal of Law & Technology 117, UCLA School of Law, Public Law Research Paper No. 21–25. Available at SSRN: https://ssrn.com/abstract=3867634
US Department of Health and Human Services. (2021). Trustworthy AI playbook. Retrieved May 30, 2022. https://www.hhs.gov/sites/default/files/hhs-trustworthy-ai-playbook.pdf
Wachter, S., Mittelstadt, B., & Russell, C. (2021). Why fairness cannot be automated: Bridging the gap between EU Non-Discrimination Law and AI. Computer Law & Security Review 41 (2021): 105567. https://doi.org/10.2139/ssrn.3547922
Wyden, R., Booker, C., & Clarke, Y. (2022). Algorithmic Accountability Act. Retrieved February 20, 2022. https://www.wyden.senate.gov/imo/media/doc/Algorithmic%20Accountability%20Act%20of%202022%20Bill%20Text.pdf
Zerilli, J., Knott, A., Maclaurin, J., & Gavaghan, C. (2018). Transparency in algorithmic and human decision-making: Is there a double standard? Philosophy & Technology, 32(4), 661–683.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The article is based on the authors’ work for BABL AI, a consultancy that focuses on responsible AI governance, algorithmic risk and bias assessments, and corporate training on responsible AI.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Hasan, A., Brown, S., Davidovic, J. et al. Algorithmic Bias and Risk Assessments: Lessons from Practice. DISO 1, 14 (2022). https://doi.org/10.1007/s44206-022-00017-z
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s44206-022-00017-z