The Assurance Measures

What it is:
Independent evaluation of AI system behavior and logic.

Why it matters:
Evaluating a proprietary system internally presents an inherent conflict of interest. While certain regulations (such as NYC Local Law 144) explicitly mandate independent audits, third-party oversight broadly provides objective verification that internal testing cannot replicate.

How we implement it:
Warden operates as an external auditing layer. We subject models to independent scrutiny to verify system behavior, ensuring results are objective and externally defensible.

What it is:
Ongoing measurement of live system outputs against defined fairness thresholds.

Why it matters:
AI systems and their underlying models evolve continuously. Point-in-time testing cannot ensure ongoing fairness. Continuous monitoring is essential for actively identifying and flagging behavioural drift or new biases over the system's operational lifecycle.

How we implement it:
We conduct audits of the production system on at least a monthly basis. For updates or new features, we also facilitate ad-hoc, pre-market audits to validate systems before they are released to the live environment.

What it is:
Evaluation of AI systems using controlled, external datasets.

Why it matters:
Relying solely on a customer's own data presents two risks: it limits the audit to the demographics already present in their applicant pool, and it is vulnerable to manipulation prior to auditing. Independent benchmarks prevent data manipulation and allow for comprehensive testing across a wider spectrum of scenarios and protected classes.

How we implement it:
Warden introduces verified, proprietary demographic profiles into the evaluation environment. This allows us to execute advanced tests (such as counterfactual analysis) and ensures audit results are statistically credible and difficult to manipulate.

What it is:
The application of two complementary AI fairness metrics to assess potential bias across protected demographic groups.

Why it matters:
Single statistical tests often miss complex discrimination patterns. For example, standard "disparate impact" testing averages outcomes across all roles; a system could heavily favor males for engineering roles and females for nursing roles, but still appear "fair" in the aggregate. Dual-method testing prevents this by looking at both group averages and individual logic.

How we implement it:
Systems are evaluated using:

• Disparate Impact Analysis (assessing equality of outcome across groups).
• Counterfactual Analysis (assessing equal treatment of individuals).

These combined measurements reveal hidden bias patterns that aggregate testing could miss.

What it is:
Aligning technical audit results with the specific requirements of relevant regulations.

Why it matters:
The regulatory landscape is fragmented, with overlapping yet distinct requirements. Organizations must understand how their system's technical performance aligns with the specific rules governing their jurisdiction.

How we implement it:
We map system outputs to evolving legal standards. We test the system against the specific protected characteristics required by California FEHA versus those required by NYC Local Law 144, providing clear alignment with relevant frameworks.

What it is:
The public sharing of independent audit results.

Why it matters:
Transparency builds fundamental trust with buyers, end-users, and candidates. Demonstrating that an AI system has undergone independent evaluation is important for market confidence and regulatory defensibility.

How we implement it:
Once each audit lifecycle is complete, Warden Assured vendors publish the results of their audits publicly (such as directly on their website or by proactively sharing them with customers) allowing stakeholders to review audit details without exposing proprietary algorithms.