Amazon's Scrapped AI Recruiting Tool: The Lessons
Amazon AI recruiting bias refers to the gender discrimination found in an internal AI hiring tool that Amazon built starting in 2014 and scrapped around 2018 after engineers found it systematically down-ranked resumes that contained the word "women's" and penalized graduates of two all-women's colleges. The tool learned bias from past hiring data dominated by male applicants, taught itself that male candidates were preferable, and could not be reliably corrected. It is the most-cited real-world example of how machine learning replicates historical discrimination when trained on biased data, and it remains a reference point for AI hiring regulation and procurement reviews today.
What was Amazon's AI recruiting tool and what did it do?
Amazon's machine-learning team began building an experimental resume-screening engine in 2014. The goal was direct: feed the system a batch of resumes and have it return the top candidates, scoring applicants from one to five stars.
The model was trained on resumes submitted to Amazon over the previous ten-year period. It learned patterns from those documents and the outcomes attached to them, then applied those patterns to new applicants. Because the training set reflected a tech-industry applicant pool and hiring history that skewed heavily male, the system inferred that maleness correlated with the candidates Amazon had historically advanced.
By 2015, the team found the tool was not rating candidates in a gender-neutral way for software developer and other technical roles. According to reporting on the project, the system:
Penalized resumes that included the word "women's" (for example, "women's chess club captain").
Downgraded applicants from at least two all-women's colleges.
Favored language more common in resumes submitted by male engineers, such as certain action verbs.
Amazon edited the system to make it neutral to those specific terms. That correction did not solve the underlying problem. The model could find other proxies for gender, and there was no guarantee it would not learn new discriminatory patterns. Amazon disbanded the team and scrapped the project around 2018. The company has stated the tool was never used to evaluate real candidates in production hiring decisions.
Why did Amazon's AI tool become biased against women?
The bias was a direct consequence of how supervised machine learning works. The model optimized to match historical decisions, and those decisions carried decades of skew.
How does training data create discrimination?
A resume-screening model learns by example. It is shown thousands of past resumes labeled with outcomes, and it identifies the features that separate the resumes that succeeded from those that did not. When the historical pool is dominated by one group, the model treats the characteristics of that group as signals of quality.
In Amazon's case, most past technical resumes came from men. The model had no concept of fairness or job performance. It had only correlation. It found that words and patterns associated with male applicants appeared more often in the resumes Amazon had advanced, so it scored those patterns higher. This is proxy discrimination: the system was not given gender as an input, but it reconstructed gender from correlated text and acted on it.
Why couldn't Amazon just remove gender from the data?
Removing an explicit gender field does not remove gender from a resume. Dozens of features carry signal about a candidate's gender, including:
College names (women's colleges, fraternities versus sororities).
Activity and club names.
Word choice and phrasing patterns that differ across demographic groups.
Sports, certifications, and affiliations.
When engineers blocked specific flagged terms, the model could still assemble gender from the remaining features. This is why redaction alone does not produce a fair model. A system trained to reproduce biased outcomes will find new proxies for the same protected characteristic after you delete the obvious ones.
What are the lessons from Amazon's scrapped AI recruiting tool?
The Amazon case produces a set of concrete lessons for any organization buying or building hiring AI. These apply whether the tool screens resumes, ranks candidates, scores video interviews, or recommends matches.
Lesson: Biased data produces biased models.
What it means: A model trained on historical decisions is likely to learn and reproduce the biases present in those decisions.
What to do: Audit training data for demographic imbalances and historical bias before model development.
Lesson: Removing protected fields is not enough.
What it means: AI models can infer protected characteristics through proxy variables even when sensitive attributes are removed.
What to do: Test for proxy discrimination rather than checking only for the presence of explicit protected attributes.
Lesson: Accuracy and fairness are separate metrics.
What it means: A model can achieve high predictive accuracy while still producing discriminatory outcomes.
What to do: Measure fairness using outcome disparities across demographic groups in addition to traditional accuracy metrics.
Lesson: Internal tools still create legal exposure.
What it means: Building an AI system in-house instead of purchasing it from a vendor does not eliminate legal responsibility.
What to do: Apply the same governance, testing, and audit standards to internally developed and third-party AI systems.
Lesson: Human review is not automatic protection.
What it means: Recruiters and decision-makers may over-rely on AI recommendations instead of exercising independent judgment.
What to do: Define clear points where humans can override AI decisions and regularly verify that meaningful oversight is taking place.
Lesson 1: Past hiring data encodes past discrimination
If a company's historical hiring favored one group, a model trained on that history will favor that group. The data is not neutral. Before training any selection model, an organization should profile the training set for representation across gender, age, and race, then decide whether that distribution is acceptable to reproduce.
Lesson 2: Test for disparate impact, not just accuracy
Amazon's tool may have been accurate at predicting which resumes resembled past hires. That is the problem, not a defense. Disparate impact measures whether a selection process passes one group at a meaningfully different rate than another. A model can score high on conventional accuracy and still fail a disparate-impact test. The two are different questions and require different checks.
Lesson 3: "We never deployed it" is not a full safeguard
Amazon caught its problem before production. Many organizations do not. The lesson is not that internal experiments are safe; it is that bias is the default outcome of training on biased data, and the only reason Amazon avoided harm was that its own engineers tested for skew and found it. That testing is the safeguard, not the decision to build internally. For a deeper treatment of how these failures surface in production screening, see our analysis of AI resume screening bias.
How does the Amazon case connect to AI hiring law today?
The Amazon tool predates most AI-specific hiring regulation, but it became the standard example that lawmakers and regulators cite when explaining why that regulation exists. The legal environment in 2026 treats automated employment decision tools as a category that requires testing and, in several jurisdictions, formal audits.
What laws now govern AI in hiring?
NYC Local Law 144 requires a bias audit of automated employment decision tools used on candidates in New York City, with enforcement that began July 5, 2023. Employers must publish a summary of audit results and notify candidates.
EEOC guidance under Title VII and the Americans with Disabilities Act (ADA) states that existing anti-discrimination law applies to algorithmic selection tools. An employer can be liable for a tool's disparate impact even when a vendor built it.
The EU AI Act classifies AI systems used in employment, including recruitment and candidate filtering, as high-risk under Annex III, which triggers obligations for risk management, data governance, and human oversight.
The Illinois Artificial Intelligence Video Interview Act, effective January 2020, requires notice and consent when AI analyzes video interviews.
Colorado's AI law (SB 24-205), signed in 2024, established duties for high-risk AI systems, including those used in employment decisions, to avoid algorithmic discrimination. Its status is in flux: a federal court stayed enforcement in April 2026, and Colorado enacted a replacement, SB 26-189, in May 2026, narrowing the original obligations. Employers in Colorado should confirm the current requirements before relying on either version.
What enforcement cases have followed?
In 2023, the EEOC settled with iTutorGroup for $365,000 after its software automatically rejected older applicants, reportedly women over 55 and men over 60. It was the agency's first AI hiring discrimination settlement.
Mobley v. Workday alleges that Workday's AI-based applicant screening discriminated on the basis of age, race, and disability. A court allowed the case to proceed, and it is being watched for how liability may attach to the vendor of a screening tool, not only the employer using it.
The connection from Amazon to these cases is consistent. A tool does not need explicit intent to discriminate to create legal exposure. Outcome disparities are enough to trigger scrutiny under existing employment law.
How can organizations avoid Amazon's mistake?
The corrective is a process, not a single fix. Organizations building or buying hiring AI in 2026 can follow a defined sequence.
Profile the training data. Measure demographic representation in the historical data before any model is trained. Document the skew you find.
Define the target variable carefully. Decide what the model is predicting. "Resembles past hires" reproduces past bias. "Performs the job well" requires real performance data, which is harder to obtain and often unavailable.
Test for proxy discrimination. Check whether the model can predict gender, age, or race from the features it uses, even after protected fields are removed.
Run a disparate-impact analysis. Calculate selection rates by group and compare them. Investigate any group that passes at a meaningfully lower rate.
Commission an independent bias audit. Where required by law (such as NYC Local Law 144), use a qualified independent auditor. Where not required, do it anyway and keep the documentation.
Set human-override rules. Specify where a human can and must overrule the system, and check that recruiters actually exercise that authority rather than deferring to the score.
Monitor after deployment. Bias can reappear as applicant pools and the model's environment change. Re-audit on a schedule, not once.
What should buyers ask AI hiring vendors?
If you are buying rather than building, the burden of proof belongs to the vendor. Ask for:
The most recent bias audit report and the name of the independent auditor.
The disparate-impact ratios by gender, age, and race for the specific use case you are buying.
A description of the training data and its demographic composition.
Documentation of proxy testing and what features the model uses.
Written terms on liability allocation if the tool produces a discriminatory outcome.
A vendor that cannot produce these is asking you to take on the risk Amazon caught in its own testing.
Frequently asked questions
Did Amazon ever use the biased AI tool to hire people?
Amazon has stated the experimental tool was never used to make actual hiring decisions and that recruiters never relied on it to evaluate candidates. The project was scrapped around 2018 after engineers concluded it could not be made reliably neutral. It is cited as a problem caught in development, not a deployed system that harmed applicants.
Why didn't removing the word "women's" fix the bias?
Blocking specific terms only removes the proxies engineers can see. A model trained to reproduce biased outcomes finds new correlates of gender in the remaining data, such as college names, club names, and phrasing patterns. The bias lived in the training data and the target variable, so editing individual words could not remove the cause.
Is using AI to screen resumes illegal?
Using AI to screen resumes is not illegal by itself. It becomes a legal problem when the tool produces a disparate impact on a protected group or violates a specific statute such as NYC Local Law 144 or the Illinois Artificial Intelligence Video Interview Act. Employers remain liable under Title VII and the ADA for discriminatory outcomes, including those produced by vendor tools.
What is proxy discrimination in AI hiring?
Proxy discrimination occurs when an AI system reconstructs a protected characteristic, such as gender or race, from other correlated data, then acts on it, even though the protected trait was never an input. Amazon's tool inferred gender from college names and word choice. Removing the explicit field does not stop a model from finding and using proxies for it.
Next steps checklist
Inventory every automated tool currently used in your hiring funnel, built and bought.
Request or commission a bias audit for each, with disparate-impact ratios by group.
Confirm your jurisdiction's requirements (NYC Local Law 144, Colorado SB 24-205 and SB 26-189, Illinois AIVIA, EU AI Act if you hire in the EU).
Add proxy-discrimination testing to your model evaluation, not just accuracy.
Write human-override rules and verify recruiters use them.
Set a recurring re-audit schedule rather than a one-time check.
