Responsible AI Toolbox, in depth: RAIInsights, error-analysis cohorts, fairness gates and analysing LLM-backed decisions

What is in the toolbox

The project ships four dashboards (the combined Responsible AI dashboard plus separate error analysis, interpretability and fairness dashboards) and an SDK. The combined dashboard is the one to learn: it is built from components you switch on individually.

Support is uneven across task types. According to the responsibleai README, explainability, error analysis and counterfactuals support binary and multi-class classification and regression, causal analysis supports binary classification and regression, and none of them support multilabel tasks. Text and vision models are handled by separate packages (responsibleai-text and responsibleai-vision), with example notebooks that include question answering on SQuAD.

Setting up RAIInsights

Everything hangs off one RAIInsights object. It takes the model, the training and test frames (with the label column still in them), the label column name and the task type. Categorical features go in a FeatureMetadata object; the older categorical_features argument is deprecated. You register analyses with add(), run them with compute(), and either open the dashboard or save the result:

from responsibleai import RAIInsights
from responsibleai.feature_metadata import FeatureMetadata
from raiwidgets import ResponsibleAIDashboard

meta = FeatureMetadata(
    categorical_features=["language", "channel", "plan"],
    dropped_features=["customer_region"],      # analysed, but not fed to the model
)

rai = RAIInsights(
    model=escalation_model,                    # sklearn-style predict / predict_proba
    train=train_df, test=test_df,              # both include the "escalate" column
    target_column="escalate",
    task_type="classification",
    feature_metadata=meta,
)

rai.error_analysis.add(max_depth=3, num_leaves=31, min_child_samples=20)
rai.explainer.add()
rai.counterfactual.add(total_CFs=10, desired_class="opposite",
                       features_to_vary=["sentiment", "wait_minutes", "prior_tickets"])
rai.compute()

rai.save("artifacts/rai/escalation-v14")       # reload with RAIInsights.load(path)
ResponsibleAIDashboard(rai)                     # interactive, in a notebook

save is an instance method and load is a static method on the class, so a review job can reload exactly what the training job computed. The model must be picklable for this to work, unless you pass a picklable serializer object with its own save and load methods; a model that wraps a network client usually is not picklable, which is the first thing that breaks when people try this with LLM-backed classifiers.

Error analysis: finding failing cohorts

Error analysis is the most useful component and the least known. It fits a shallow surrogate decision tree whose target is not the label but whether the model got each test row wrong. Each leaf is a cohort described by a conjunction of feature conditions, such as language != en AND message_words <= 20, annotated with its error rate and its share of all errors. A heat map view does the same for one or two chosen features.

This turns a vague question (where is the model bad?) into a ranked list of concrete slices you can inspect, relabel or collect data for. The parameters matter: max_depth limits how many conditions a cohort can have, and min_child_samples stops the tree from carving out tiny cohorts whose error rate is noise. Keep cohorts large enough that a difference in error rate is statistically meaningful before anyone acts on it; 20 rows is a floor, not a target.

Read each cohort on two axes. Error rate says how badly the model does inside the slice; error coverage says what share of all mistakes the slice holds. A cohort with a 60% error rate and 0.5% coverage is a curiosity unless the people in it face serious harm, while one with a 12% error rate and 30% coverage is usually where retraining pays off first. Save the cohort definitions you act on: the dashboard lets you name cohorts and carry them into the other views, and the same filters become regression tests that must not get worse on the next model version.

Fairness as a release gate

The dashboard shows fairness metrics interactively, which is right for a review meeting and wrong for a release gate. For the gate, call Fairlearn directly; it is the same metric code, without a UI:

from fairlearn.metrics import MetricFrame, false_negative_rate, selection_rate
from sklearn.metrics import accuracy_score

y_pred = escalation_model.predict(X_test)
mf = MetricFrame(
    metrics={"accuracy": accuracy_score, "fnr": false_negative_rate,
             "selection": selection_rate},
    y_true=y_test, y_pred=y_pred,
    sensitive_features=test_df["language"],
)
print(mf.by_group)
gap = mf.difference()["fnr"]
if gap > 0.05:                                  # threshold agreed with the risk owner
    raise SystemExit(f"FNR gap across languages is {gap:.3f}; release blocked")

Choose the metric from the harm. For an escalation model a false negative means a customer who needed a human did not get one, so the false-negative-rate gap is the gate; selection-rate parity would be the wrong target. AI Fairness, in depth explains why the criteria conflict and how to mitigate at data, training and threshold stages; Fairlearn's ThresholdOptimizer implements the threshold option.

Explanations, counterfactuals and causal analysis

The explainer gives global feature importance and per-row explanations. Use it to check that the model relies on what you expect: if customer_region was dropped but a proxy such as postcode dominates the importances, the drop did nothing. Explanations describe the model, not the world, so a high importance is evidence of reliance, not of a causal effect.

Counterfactuals answer the question a reviewer actually asks about one decision: what would have to change for the outcome to flip? Restrict features_to_vary to things that can change and set permitted_range to realistic values, otherwise DiCE will happily suggest changing a customer's language or age. A counterfactual that requires changing a protected attribute is itself a finding.

Causal analysis estimates the effect of intervening on chosen treatment features using EconML. It answers policy questions (would reducing wait time reduce escalations?) but rests on the usual observational assumptions, chiefly that the confounders are in the data. Treat its output as a hypothesis for an experiment.

Analysing an LLM-backed decision

To analyse an LLM-backed decision, wrap it in a class with the scikit-learn interface. The toolbox calls predict and predict_proba many times, especially for counterfactuals and explanations, so cache responses and pin the model version and temperature, or the analysis is both expensive and non-reproducible:

import numpy as np

class LlmEscalationClassifier:
    """Sklearn-style wrapper so RAIInsights can analyse an LLM decision."""
    classes_ = np.array([0, 1])

    def __init__(self, client, model_id, prompt_template):
        self.client, self.model_id, self.tmpl = client, model_id, prompt_template
        self.cache = {}                          # row -> score; pickled with the instance

    def _p_escalate(self, row_key):
        if row_key not in self.cache:
            reply = self.client.classify(self.model_id, self.tmpl.format(*row_key), temperature=0)
            self.cache[row_key] = float(reply.probability)   # however your client exposes a score
        return self.cache[row_key]

    def predict_proba(self, X):
        p = np.array([self._p_escalate(tuple(r)) for r in X.itertuples(index=False)])
        return np.column_stack([1 - p, p])

    def predict(self, X):
        return (self.predict_proba(X)[:, 1] >= 0.5).astype(int)

The client.classify call stands in for whatever API you use. Before saving, swap the live client for a replay of cached responses so the artifact is picklable and repeatable. For free-text inputs, derive tabular features (language, length, topic, extracted sentiment) so cohorts are readable, and keep the raw text alongside for inspection.

Worked example

Consider the escalation classifier above, trained on 120,000 historical tickets, with a 4% overall error rate. The numbers below are illustrative, but the shape is typical. Error analysis with depth 3 finds that tickets with language != en and message_words <= 20 make up 6% of the test set but 27% of errors, with an 18% error rate. The Fairlearn gate shows a false-negative rate of 9% for non-English tickets against 3% for English, failing a 5-point threshold.

The data explorer shows non-English tickets are 8% of training data and are labelled by a different team with a lower escalation base rate. Explanations show the model leans on an LLM-extracted sentiment score, which turns out to be weaker on short non-English messages. The fixes follow directly: audit the second team's labels, collect more short non-English tickets, compute sentiment in the source language, and recompute. The saved artifact from each run goes into the model's documentation, described in Model Cards, in depth.

Operating it

• Pin versions. As of this check, the latest responsibleai and raiwidgets release on PyPI is 0.36.0, published in July 2024. Pin it, check it supports your Python version, and test upgrades on a saved artifact.
• Run headless in CI. Compute insights and Fairlearn gates in the training pipeline, save the artifact, and open the dashboard only for review.
• Version the artifact with the model. Store it next to the model weights and data snapshot so an auditor can reproduce the view that approved a release.
• Assign owners. A cohort finding is only useful if someone owns the fix; route findings through the process in AI Governance Program Structure.

Failure modes

• Test-set truncation. RAIInsights has maximum_rows_for_test defaulting to 5,000. Larger test sets trigger a warning, and the insights are computed on the first 5,000 rows, not a random sample; the full set is used only for side computations such as feature ranges. If your test frame is sorted by date or region, the cohorts you see are biased. Shuffle or stratify before passing it in.
• Sensitive features fed to the model by accident. Mark them as dropped features so they are analysed but not used, and check importances for proxies.
• Tiny cohorts. A 40% error rate on 12 rows is noise. Report counts with every rate.
• Explanations read as causes. Feature importance says what the model uses, not what drives the outcome in the world.
• Unrealistic counterfactuals. Without constraints DiCE proposes impossible changes.
• Non-deterministic wrapped models. An LLM sampled at non-zero temperature gives a different analysis on every run.

Trade-offs

The toolbox's strength is integration: one object and one view connect where the model fails, who it fails, why and what would change it. The cost is a heavy dependency tree, a tabular-first design and an interactive tool that does not by itself gate anything. Using Fairlearn, an explainer library and your own slicing code separately is lighter and easier to embed in CI, but loses the joined-up workflow that makes review meetings productive. Many teams use both: plain libraries for gates, the dashboard for investigation.

What to do next

1. List every decision point in your LLM system that produces a label or score, and name the harm of each error type.
2. For one of them, build a test frame with the label, model output, tabular features and sensitive attributes; shuffle it.
3. Run RAIInsights with error analysis and the explainer, and read the top three error cohorts with their counts.
4. Add a Fairlearn MetricFrame gate on the error rate that matches the harm, with a threshold agreed with the risk owner.
5. Save the artifact next to the model version and link it from the model card.
6. Re-run on every retrain and compare cohorts across versions.