Five AI HR Failures and What They Teach
AI in HR risks are the legal, ethical, and operational exposures that arise when organizations use automated employment decision tools for sourcing, screening, ranking, and interviewing candidates. The five most-cited failures fall into recurring categories: biased training data, proxy discrimination against protected classes, disparate impact on age and disability, vendor liability when a third-party tool screens applicants, and regulatory noncompliance with bias-audit and notice laws. The lesson across all five is the same: AI does not remove human bias from hiring. It scales and hides that bias unless you measure, audit, and govern the system.
The cases below are drawn from public litigation, regulator actions, and company disclosures. Each one maps to a specific control that HR and data teams can put in place now.
What are the biggest AI in HR risks?
The biggest risks are not abstract. They show up as lawsuits, enforcement actions, and abandoned multi-year projects. The pattern is consistent across the documented record:
Historical data encodes historical discrimination. A model trained on past hiring decisions learns to reproduce them.
Removing a protected attribute does not remove the bias. Models reconstruct gender, age, race, and disability from correlated signals, or proxies.
Liability follows the decision, not the software license. Employers and, increasingly, vendors are named in suits even when the model is third-party.
The law is already active. Bias-audit, notice, and high-risk classification requirements are in force in multiple jurisdictions, and enforcement has already produced settlements.
The table below summarizes the five failures and the control each one teaches.
Failure: Amazon recruiting tool.
What went wrong: The model down-ranked resumes associated with women.
Control it teaches: Audit training data for historical bias before deployment.
Failure: iTutorGroup (EEOC).
What went wrong: The software automatically rejected older applicants.
Control it teaches: Test for age-related disparate impact and remove hard age cutoffs.
Failure: Mobley v. Workday.
What went wrong: AI screening was alleged to discriminate based on age, race, and disability.
Control it teaches: Define vendor liability and document the full decision chain.
Failure: Proxy discrimination pattern.
What went wrong: Neutral-looking features can act as stand-ins for protected traits.
Control it teaches: Run proxy analysis instead of relying only on protected-attribute removal.
Failure: Regulatory noncompliance.
What went wrong: AI tools were deployed without required audits or candidate notices.
Control it teaches: Map every AI hiring tool to applicable requirements, including NYC Local Law 144, the EU AI Act, and relevant state laws.
Why does AI in hiring reproduce human bias?
AI screening tools learn patterns from data. When the training data reflects a workforce shaped by decades of biased human decisions, the model treats that bias as the target outcome. It optimizes to predict who looked like a successful hire in the past, which means it favors the demographics that were historically hired and promoted.
Three mechanisms drive this:
Label bias. The "good hire" labels in the data come from human decisions that were themselves biased. The model inherits the bias as ground truth.
Feature proxies. Even with name, gender, and age stripped out, features like the name of a women's college, a graduation year, gaps in employment, or membership in certain clubs correlate with protected characteristics.
Feedback loops. A model that screens out a group reduces that group's representation in future hires, which then trains the next model on even more skewed data.
Failure 1: Amazon's recruiting engine and biased training data
Amazon built an internal AI recruiting tool to score resumes and reportedly scrapped it around 2018 after engineers found it penalized resumes that included signals associated with women, including the word "women's" and certain all-women colleges. The system had been trained on resumes submitted to the company over a prior multi-year period, a pool dominated by men in technical roles. The model learned that male-associated patterns predicted "success" and down-ranked the rest.
The lesson: Audit your training data for historical skew before you train anything. If your past hires are demographically lopsided, a model trained on them will encode that lopsidedness as a hiring signal. Attribute removal alone does not fix it, because the model reconstructs the signal from proxies. For a deeper breakdown of this case and the controls it implies, see the lessons from Amazon's AI recruiting tool.
Failure 2: iTutorGroup and age disparate impact
In 2023, the EEOC settled a case against tutoring company iTutorGroup for $365,000. The agency alleged the company's recruiting software was configured to automatically reject female applicants age 55 or older and male applicants age 60 or older. This was the first EEOC settlement involving AI-driven hiring discrimination.
What makes this case instructive is that the discrimination was not a subtle statistical artifact. It was a configured rule that produced a clear disparate impact on older workers, a protected class under the Age Discrimination in Employment Act (ADEA). The software did exactly what it was told. The failure was governance: no one tested the configuration against anti-discrimination law before it went live.
The lesson: Test every automated screening rule and model output for disparate impact on age and other protected classes. Remove hard demographic cutoffs entirely. A tool that filters on age, even indirectly through a proxy like graduation year, creates direct legal exposure.
Who is liable when a third-party AI tool discriminates?
Liability is the question that worries HR leaders and general counsel most. The traditional assumption was that the employer who makes the final decision carries the risk. Recent litigation is testing whether the vendor that builds and operates the screening model can also be held liable.
Failure 3: Mobley v. Workday and vendor liability
Mobley v. Workday is a closely watched case in which a plaintiff alleged that Workday's AI-based applicant-screening tools discriminated against applicants on the basis of age, race, and disability. The significance is procedural as much as substantive: a federal court allowed the theory to proceed that an AI vendor could be treated as an agent of the employers using its software, exposing the vendor itself to discrimination claims. The court has since allowed an age-discrimination collective action to move forward.
This reframes the risk model for every company that licenses third-party hiring AI:
The employer remains responsible for the outcomes of tools it deploys.
The vendor may also be liable as an agent, depending on how the case law develops.
Contracts do not transfer legal duty under anti-discrimination statutes the way they transfer commercial risk.
The lesson: Map the full decision chain. Document which system touches which candidate at which stage, who configured it, and what data it used. Require vendors to produce bias-audit results and to contractually support your compliance obligations. Do not assume a license agreement insulates you from Title VII, the ADA, or the ADEA.
Failure 4: Proxy discrimination as a pattern, not an event
Proxy discrimination is the failure mode that connects the others. It happens when a model uses a facially neutral feature that correlates strongly with a protected characteristic. The model never reads race or gender directly, yet it reproduces the disparity.
Documented and well-understood proxies in hiring data include:
Zip code or home address, which correlates with race and national origin in many regions.
Graduation year or years of experience, which correlates with age.
Employment gaps, which correlate with disability, caregiving, and gender.
Names of schools, clubs, or affiliations associated with a single gender or ethnic group.
Language and phrasing patterns in resumes that differ across demographic groups.
The EEOC's technical guidance on AI under Title VII makes clear that an employer can be liable for disparate impact even when a tool was not designed to discriminate and even when the discrimination operates through a proxy. Intent is not required for disparate-impact liability.
The lesson: Attribute removal is necessary but not sufficient. Run a dedicated proxy analysis: measure the correlation between every input feature and protected characteristics, and measure the model's outcomes across demographic groups. The four-fifths rule is a common starting threshold. If a feature drives disparate outcomes without job-related justification, remove it or constrain the model.
What laws govern AI in hiring?
Regulation moved from proposal to enforcement faster than most HR teams adjusted. Deploying a hiring tool without mapping it to the applicable laws is now its own category of failure.
Failure 5: Regulatory noncompliance with active AI hiring laws
Several laws now impose specific, auditable obligations on automated employment decision tools. The most-cited are below.
Law / Rule: NYC Local Law 144
Jurisdiction: New York City.
Core obligation for hiring AI: Conduct an independent bias audit of Automated Employment Decision Tools (AEDTs) and provide advance notice to candidates before use.
Status: Enforcement began in July 2023.
Law / Rule: EEOC AI guidance
Jurisdiction: United States (federal).
Core obligation for hiring AI: Ensure AI hiring tools comply with Title VII disparate impact requirements and ADA reasonable accommodation obligations. Employers remain responsible for AI-assisted employment decisions.
Status: Active technical guidance.
Law / Rule: EU AI Act
Jurisdiction: European Union.
Core obligation for hiring AI: Classifies employment and recruitment AI as high-risk under Annex III, requiring conformity assessment, technical documentation, risk management, transparency, and human oversight.
Status: Compliance obligations are being phased in.
Law / Rule: Illinois AI Video Interview Act
Jurisdiction: Illinois.
Core obligation for hiring AI: Provide notice, obtain candidate consent, and explain how AI analyzes recorded video interviews.
Status: Effective January 2020.
Law / Rule: Colorado AI Act (SB 24-205)
Jurisdiction: Colorado.
Core obligation for hiring AI: Requires organizations to exercise reasonable care to prevent algorithmic discrimination in high-risk AI systems, including those used for employment decisions.
Status: Enacted; the effective date has been delayed and should be verified against the latest legislative updates.
The lesson: Maintain a current inventory of every tool that touches a hiring decision and map each one to the laws of every jurisdiction where you recruit. NYC Local Law 144 requires an independent bias audit and candidate notice for covered tools. The EU AI Act treats hiring AI as high-risk, which carries documentation, transparency, and human-oversight duties. The Illinois Artificial Intelligence Video Interview Act requires notice and consent before AI analyzes a recorded interview. State laws such as the Colorado AI Act continue to shift in effective date and scope, so treat compliance as a standing program, not a one-time legal review.
How can HR teams reduce AI in HR risks?
The five failures converge on a short set of controls. None of them require abandoning AI in hiring. They require governing it.
Build a tool inventory. List every system that sources, screens, ranks, or interviews candidates, including features inside your ATS and HRIS.
Audit training data before deployment. Check for demographic skew in the historical data the model learns from. Skewed inputs produce skewed outputs.
Run proxy analysis, not just attribute removal. Measure correlations between features and protected characteristics, and measure outcomes across groups.
Test for disparate impact continuously. Use thresholds like the four-fifths rule, monitor in production, and retest after every model or configuration change.
Define vendor liability in writing. Require bias-audit documentation, audit rights, and compliance support in contracts. Assume the legal duty stays with you regardless.
Map every tool to applicable law. Cover NYC Local Law 144, EEOC guidance, the EU AI Act, the Illinois AI Video Interview Act, the Colorado AI Act, and any new state rules.
Keep a human in the loop with real authority. Document who reviews flagged decisions and how candidates can request human review or accommodation.
Next Steps checklist
Step: Inventory all AI hiring tools and Applicant Tracking System (ATS) features.
Owner: HR Operations and IT.
Output: A living register of all Automated Employment Decision Tools (AEDTs).
Step: Conduct a historical bias audit of the training data.
Owner: Data and AI team.
Output: A documented analysis of demographic representation and potential bias.
Step: Perform proxy discrimination and disparate impact testing.
Owner: Data and AI team, together with Legal.
Output: A report containing four-fifths rule results and proxy correlation analysis.
Step: Obtain independent bias audits from every AI vendor.
Owner: Procurement and Legal.
Output: Bias audit documentation ready for compliance with NYC Local Law 144.
Step: Map each AI tool to the applicable laws in every hiring jurisdiction.
Owner: Legal and Compliance.
Output: A compliance matrix listing applicable regulations, audit requirements, and notice deadlines.
Step: Define human review and accommodation procedures for AI-assisted decisions.
Owner: HR and Legal.
Output: A documented escalation process and accommodation policy.
Step: Establish a recurring AI re-audit schedule.
Owner: Compliance.
Output: A calendar of periodic audits aligned with model updates and material changes.
FAQ
Does removing names and gender from resumes prevent AI bias?
No. Removing protected attributes is necessary but not enough. Models reconstruct gender, age, race, and disability from correlated features such as school names, graduation years, employment gaps, and zip codes. This is proxy discrimination. You have to run proxy-correlation analysis and test outcomes across demographic groups, not just delete the obvious fields.
Is an employer liable if a third-party AI tool discriminates?
Generally yes. Under anti-discrimination statutes like Title VII, the ADA, and the ADEA, the employer that uses a tool is responsible for its outcomes. Mobley v. Workday is also testing whether the AI vendor itself can be held liable as an agent. A license agreement does not transfer your legal duty, so require bias audits and compliance support from vendors.
What does NYC Local Law 144 require?
NYC Local Law 144 requires employers using covered automated employment decision tools to commission an independent bias audit within the prior year, publish a summary of the results, and provide notice to candidates. Enforcement began in July 2023. It applies to roles where candidates are screened by qualifying tools in New York City.
Can AI be used in hiring legally and responsibly?
Yes. AI in hiring is legal in most jurisdictions when governed properly: audit training data, run proxy and disparate-impact testing, keep documentation, provide candidate notice where required, and maintain human oversight with real authority. The failures above came from missing governance, not from the existence of the technology itself.
