Can AI Be Fair? Tackling the Deep Roots of Data Bias

When Data Reflects Historical Inequities
AI systems learn patterns from data, but that data is rarely neutral. Historical records, whether police reports, credit scores, or hiring histories, often reflect decades of inequality. When machine learning models use this data to make decisions, they risk amplifying past injustices rather than correcting them. For example, predictive policing systems trained on historical arrest data have disproportionately targeted minority neighborhoods, not necessarily because crime is higher, but because enforcement has been heavier in those areas historically1.
Similar issues have emerged in facial recognition. Studies by the National Institute of Standards and Technology (NIST) found that many commercial facial recognition algorithms have higher false positive rates for African American and Asian faces compared to white faces2. These biases are not the result of malicious intent but arise from imbalanced training datasets that overrepresent certain demographics. When deployed in real-world contexts like airport screening or police investigations, such systems can lead to unjust outcomes for individuals who are already marginalized.
How Incomplete Data Skews Decision-Making
AI does not fill in knowledge gaps with empathy or context. When data is missing or underrepresentative, models draw conclusions from incomplete pictures. For instance, hiring algorithms trained on resumes from predominantly male applicants have been shown to downgrade resumes featuring terms like "women’s chess club" or graduates from women’s colleges. This happened because the algorithm associated past hiring success with male-dominated profiles3.
These gaps are not limited to gender. Healthcare algorithms have underestimated the needs of Black patients because historical data reflected lower healthcare expenditure on those populations, not lower need. A study published in Science found that algorithmic bias in healthcare risk prediction tools led to Black patients receiving fewer services than equally sick white patients4. These examples point to a critical lesson: AI systems are only as fair as the data they are built on, and data gaps often mirror systemic neglect.
Human Choices Shape AI Behavior
Beyond data, humans make design choices that can embed bias into algorithmic systems. These include how outcomes are defined, which features are selected, and how models are trained and tested. If a hiring algorithm optimizes for "culture fit," it may inadvertently prioritize candidates who resemble current employees, thus perpetuating homogeneity. If a city’s traffic management AI is optimized solely for flow efficiency, it may route more congestion through lower-income neighborhoods without considering environmental justice impacts.
Transparency in these design choices is often lacking. Proprietary systems developed by vendors are frequently deployed in municipalities without local officials fully understanding how decisions are made. Without access to the algorithm’s inner workings or the ability to question its logic, public administrators struggle to ensure fairness. This opacity limits accountability and undermines public trust, particularly when outcomes affect access to housing, employment, or public safety.
Embedding Ethical Frameworks in Practice
To counteract these risks, organizations are adopting ethical frameworks focused on fairness, transparency, and accountability. Fairness involves identifying and mitigating disparate impacts across demographic groups. This may include disaggregated testing of model outputs by race, gender, age, or geography. Transparency means making models explainable to non-technical stakeholders, enabling public scrutiny. Accountability refers to establishing clear lines of responsibility for AI decisions and ensuring that human oversight remains in place.
Government agencies and academic institutions are developing toolkits for ethical AI governance. The U.S. Government Accountability Office has issued guidance on AI accountability that includes practices like third-party audits, bias testing, and documentation of design decisions5. The OECD’s AI Principles, adopted by over 40 countries, call for AI systems that are inclusive and respect democratic values6. These frameworks are not one-size-fits-all but provide a structure for local governments and organizations to build their own governance models based on community needs.
Practical Steps for Responsible AI Use
Municipal leaders and administrators do not need to be data scientists to advocate for ethical AI. One practical step is to require explainability in any AI system procured. This means vendors must provide clear documentation on how the system works, what data it uses, and how decisions are made. Explainable AI helps decision-makers assess whether the system aligns with public values and legal standards before deployment.
Another action is to implement ethical audits for high-stakes AI applications, particularly those affecting public services, law enforcement, or resource allocation. These audits should evaluate both technical performance and social impact, ideally involving community stakeholders. Finally, keep humans in the loop. Even the most advanced AI should support, not replace, human judgment—especially when the consequences affect people's rights or well-being. Requiring human review before final decisions are made adds a layer of accountability that no algorithm can replicate.
Building a Learning Culture Around AI
For AI to serve the public interest, local governments must foster a culture of continuous learning. This includes training staff to understand the basics of algorithmic decision-making, encouraging interdepartmental collaboration on AI projects, and engaging with academic and community partners. Cities like New York and Amsterdam have started public AI registries that disclose where and how AI is used in municipal services. These efforts promote transparency and allow residents to participate in shaping how technology is applied in their communities7.
Regular review processes should also be established. As data and societal norms evolve, AI systems need recalibration. What was fair five years ago may no longer be acceptable today. Embedding review cycles into procurement and performance management processes ensures that AI tools remain aligned with community values. By treating AI governance as a living process rather than a one-time checklist, municipal leaders can better navigate the ethical complexities of this rapidly advancing technology.
Bibliography
AI Now Institute. "Algorithmic Policing and Surveillance in U.S. Cities." AI Now Reports, 2019.
Grother, Patrick, Mei Ngan, and Kayee Hanaoka. "Face Recognition Vendor Test (FRVT) Part 3: Demographic Effects." NIST Interagency Report 8280, National Institute of Standards and Technology, 2019.
Dastin, Jeffrey. "Amazon Scraps Secret AI Recruiting Tool That Showed Bias Against Women." Reuters, October 10, 2018.
Obermeyer, Ziad, et al. "Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations." Science 366, no. 6464 (2019): 447–453.
U.S. Government Accountability Office. "Artificial Intelligence: An Accountability Framework for Federal Agencies and Other Entities." GAO-21-519SP, June 2021.
Organisation for Economic Co-operation and Development. "OECD Principles on Artificial Intelligence." 2019.
City of Amsterdam. "Algorithm Register." Accessed March 2024. https://algoritmeregister.amsterdam.nl/en.





Join the discussion
Members can read and add comments on every article.