Who’s in the Data? Bringing Equity and Reflection into AI Design

Ethical reflection in AI design is not a theoretical exercise - it is a critical step in creating systems that align with democratic values and public accountability. From the earliest stages of data collection and model training, developers must ask whose experiences are represented, whose are excluded, and what assumptions are being codified. For example, an algorithm designed to predict student performance may unintentionally penalize students from under-resourced schools if it relies heavily on historical test scores without contextualizing systemic disparities. These design decisions, though often technical in appearance, carry profound ethical implications.

Instituting ethical reflection requires structured practices. Techniques such as ethics checklists, stakeholder reviews, and bias audits can help teams surface potential risks before deployment. The U.S. Government Accountability Office has recommended integrating responsible AI practices into federal acquisition and development processes, including stress-testing models for fairness and transparency across population subgroups¹. Municipal leaders and administrators can adopt similar frameworks by requiring vendors to disclose datasets used, explain model decisions in plain language, and demonstrate how potential harms have been minimized. Building these expectations into procurement policies ensures ethical concerns are addressed early and not as afterthoughts.

Learning from Public Sector Use Cases

Real-world applications of AI in government offer valuable lessons about both opportunity and risk. Predictive policing tools, for instance, have been deployed in several cities to forecast where crimes are likely to occur. However, investigations have shown that such systems can perpetuate racial bias by relying on historical arrest data, which may reflect discriminatory policing patterns rather than actual crime rates². These examples reveal how AI can amplify existing inequities if not rigorously evaluated for fairness and accountability.

Similarly, automated decision systems used in social services, such as eligibility screening for public benefits, have faced scrutiny over transparency and error rates. In Indiana, a statewide automation program that flagged citizens for fraud resulted in thousands of wrongful terminations of Medicaid and food aid benefits³. These incidents demonstrate the importance of maintaining human oversight, establishing appeals processes, and ensuring citizens have access to explanations and recourse. AI should support, not replace, equitable service delivery and due process.

The Role of Diverse Stakeholders in Shaping AI

Inclusive development is essential to ensuring AI reflects the needs and values of all communities. Too often, technical teams operate in isolation, with limited input from educators, community advocates, public administrators, or those directly impacted by the systems. Bringing diverse voices into the design process helps surface blind spots, challenge assumptions, and build more contextually appropriate tools. For instance, involving housing advocates in the development of tenant screening algorithms can help identify criteria that may disproportionately affect low-income renters or communities of color.

Local governments can facilitate participatory des