Bias in AI hiring tools is a serious and legitimate concern. Early AI systems amplified historical biases, sometimes dramatically, by learning patterns from tainted data. Legislative bodies in New York City, Illinois, Colorado, and the EU have responded with laws requiring disclosure, auditing, and in some cases, pre-deployment bias testing.
We welcomed this scrutiny — and we commissioned an independent third-party audit of Braintrust AIR to verify what we believed about our own system. Here's what the audit found.
The Audit Process
The audit was conducted by an independent firm with no prior relationship with Braintrust. Auditors were given full access to AIR's scoring methodology, rubric design, training data documentation, and a dataset of historical interview evaluations across a demographically diverse candidate pool.
The evaluation assessed two primary questions: Does AIR produce statistically significant disparities in scoring outcomes across demographic groups? And does the overall system architecture comply with applicable regulatory frameworks, including GDPR, EEOC guidelines, and NYC Local Law 144?
The Findings
The auditors found zero adverse impact across every demographic group tested — including race, gender, national origin, and age.
The auditors specifically noted that AIR's rubric-based, transcript-driven evaluation architecture was a key factor in this outcome. By evaluating the semantic content of what candidates say — scored strictly against predefined competency criteria — rather than relying on subjective human judgment or unscientific visual or acoustic analysis, the system avoids the primary vectors through which AI systems introduce bias.
Regulatory Compliance
Review the audit's scope and date with counsel to determine how it applies to your deployment. Under NYC Local Law 144, the employer must post the audit results summary on its own website; a vendor's audit report does not discharge that duty.
What This Means for Enterprise Buyers
If your legal team or procurement process requires documented evidence of bias testing as a condition of deployment, we have it. If your jurisdiction requires an annual bias audit for automated employment decision tools, ask us for the audit documentation and confirm with counsel how it applies to your deployment.
We believe rigorous independent auditing should be standard practice for any AI system used in high-stakes decisions. We publish our results because we think that transparency builds trust — and because the data supports our confidence in AIR's fairness.
Book a demo to request the full audit report and discuss compliance requirements specific to your jurisdiction and industry.
