AI Impact Analysis, in depth: blast radius, exposure windows, bounded harm per dimension and a max-dominant aggregation rule

Three quantities behind every impact

Impact is a function of three quantities, and it helps to keep them separate. The blast radius is the set of users, transactions and downstream systems that a failure can reach. The exposure window is how long the failure keeps reaching them: the time until someone notices, plus the time to stop it. The harm per unit is what each affected user or transaction suffers. Multiply them and you have impact on one dimension.

LLM systems make all three harder to guess than in classic software. Outputs are probabilistic, so a failure often affects a fraction of requests rather than all of them. Outputs are fluent, so wrong answers do not look broken and detection is slow. And outputs are increasingly actions, through tools and agents, so one bad completion can propagate into refunds, emails, tickets and code. A scenario description should therefore always name the trigger, the fraction of traffic affected, the action surface, and who would notice first.

Architecture

The output is a small structured record per scenario, not a document. That matters because impact changes whenever the system changes: a new tool, a new consumer of the model output, a new market. If the record is data, you can recompute it in CI when the dependency graph changes, exactly like a risk register entry, which is described in the AI risk register.

Blast radius from the consumer graph

Start from the component that fails and walk outward through everything that consumes its output. Each edge carries a fraction: what share of the upstream output flows to that consumer. A support assistant might feed a refund tool on 6 percent of conversations, a ticket summariser on all of them, and a weekly analytics job that nobody reads in real time. The blast radius is the set of reachable nodes with the cumulative fraction on each path, and the nodes that touch people or money are where harm materialises.

from collections import deque

# consumer graph: node -> list of (downstream node, fraction of upstream output it receives)
GRAPH = {
    "support_llm":    [("reply_to_user", 1.0), ("refund_tool", 0.06), ("ticket_summary", 1.0)],
    "refund_tool":    [("payments", 1.0)],
    "ticket_summary": [("crm", 1.0), ("weekly_report", 1.0)],
}
HARM_NODES = {"reply_to_user", "payments", "crm"}   # where people or money are touched

def blast_radius(root, graph=GRAPH):
    """Return {node: max fraction of root traffic that can reach it}."""
    reach = {root: 1.0}
    queue = deque([root])
    while queue:
        node = queue.popleft()
        for nxt, frac in graph.get(node, []):
            f = reach[node] * frac
            if f > reach.get(nxt, 0.0):
                reach[nxt] = f
                queue.append(nxt)
    return reach

r = blast_radius("support_llm")
print({n: f for n, f in r.items() if n in HARM_NODES})
# {'reply_to_user': 1.0, 'payments': 0.06, 'crm': 1.0}

Keep the graph in the repository next to the service definitions, so adding a consumer is a reviewed change that also updates impact. The most common surprise this walk produces is a quiet consumer: a summary that lands in a CRM, gets copied into a customer email by a human, and turns a hallucination into a commitment.

The exposure window

The exposure window is time to detect plus time to repair. Time to detect is not how quickly you could notice in principle; it is how quickly the slowest honest path to noticing works today. If the only check on refunds is a daily finance reconciliation that is reviewed on weekdays, a failure that starts on a Friday evening has a realistic detection time of days. Write both a likely and a worst value, and for the worst value assume the detector you rely on is itself late.

Affected units then follow directly: units per day reaching the harm node, times the fraction of them the failure actually corrupts, times the window in days. Because the window multiplies everything, the curve below is the most useful picture in the whole exercise.

Dimensions and bounds

Harm does not live on one axis, and converting everything into money hides the cases that matter most. Use a fixed set of dimensions, each with its own unit and its own severity bands, and never add across them.

For each dimension record a likely bound and a worst bound, each with its assumptions written beside it. Worst means a plausible bad case your team would not be embarrassed to defend, not a cosmic catastrophe; if the worst case needs three independent failures, it belongs in a separate scenario. Ranges should come from evidence where you have it: incident history, red-team results on the failure mode, traffic logs for the fraction of requests that hit the trigger.

Aggregating without hiding harm

Once each dimension has a band, you need one headline. The rule that survives contact with reality is max-dominant: the scenario takes the highest band reached on any dimension, and the record keeps every dimension visible so nobody has to reverse-engineer the headline. Additive rules let five minor harms outvote one serious safety harm, which is precisely backwards, and multiplicative rules invent precision you do not have.

from dataclasses import dataclass, field

BANDS = ["negligible", "minor", "moderate", "major", "severe"]
# upper bounds per band; financial in USD, others in their own unit
THRESHOLDS = {
    "financial":   [1_000, 10_000, 100_000, 1_000_000],
    "operational": [8, 80, 400, 2_000],          # staff hours
    "trust":       [10, 500, 10_000, 100_000],   # customers who saw it
}

def band(dim, value):
    if dim == "safety":                          # any credible injury is at least major
        return "negligible" if value == 0 else ("major" if value < 5 else "severe")
    if dim == "legal":                           # count of notifiable events
        return "negligible" if value == 0 else ("moderate" if value == 1 else "major")
    for i, limit in enumerate(THRESHOLDS[dim]):
        if value < limit:
            return BANDS[i]
    return BANDS[-1]

@dataclass
class Scenario:
    name: str
    units_per_day: float          # traffic reaching the harm node
    fraction: tuple               # (likely, worst) share of that traffic corrupted
    days_to_detect: tuple         # (likely, worst)
    days_to_repair: tuple         # (likely, worst)
    harm_per_unit: dict = field(default_factory=dict)  # dim -> (likely, worst)

    def affected(self, i):
        return self.units_per_day * self.fraction[i] * (self.days_to_detect[i] + self.days_to_repair[i])

    def bounds(self):
        out = {}
        for case, i in (("likely", 0), ("worst", 1)):
            n = self.affected(i)
            dims = {d: band(d, n * h[i]) if d not in ("safety", "legal") else band(d, h[i])
                    for d, h in self.harm_per_unit.items()}
            out[case] = {"affected": round(n), "bands": dims,
                         "headline": max(dims.values(), key=BANDS.index)}
        return out

Safety and legal use event-style bands because their harm is not proportional to volume: one person injured or one notifiable breach is already serious. The thresholds above are examples; your organisation sets them once, writes them down with the risk appetite, and does not renegotiate them per scenario.

Worked example: a refund tool after an upgrade

Take a customer-support assistant with a refund tool. It handles 40,000 conversations a day and calls the refund tool in 6 percent of them, so 2,400 refund decisions a day reach payments. The scenario: after a model upgrade, a particular phrasing leads the assistant to approve refunds above the policy cap. Red-team replay of last month's transcripts suggests 2 percent of refund conversations are affected, and a plausible worst case is 10 percent if the phrasing spreads on a forum.

Operational impact is the clawback work, roughly ten minutes per case: 20 staff hours likely (minor), 400 worst (major). Trust is the 2,400 customers who later receive a reversal email in the worst case (moderate). There is no safety or legal dimension here, so the headline is minor likely and major worst. Notice where the worst case comes from: nine days of detection, not the failure rate. Add a key risk indicator that alerts when any approved refund exceeds the cap, reviewed within a day, and the worst window falls to two days: 2,400 x 0.10 x 2 = 480 affected decisions and $86,400, which is moderate. The same money spent on prompt hardening might halve the fraction; the alarm cuts the window by a factor of five. Monitoring of this kind is covered in AI risk monitoring.

Impact analysis for changes

Impact analysis is also the right lens for changes, not just failures. Before a model upgrade, a new tool or a new consumer goes live, rerun the blast-radius walk on the changed graph and diff it against the current one. A new edge into payments, email or code execution should block the release until its scenarios have bounds. A model swap that keeps the graph unchanged still changes the fractions, so replay a fixed transcript set on the new model and update the likely fraction from measurement rather than hope. Wire this into the release pipeline: the graph file and the scenario file live in the repository, the CI job recomputes bounds, and any headline that moves up a band pages the risk owner.

Failure modes

• Average-case severity. Rating the typical failure and calling it the impact. Always carry a worst bound; tails are where the money and the injuries are.
• Dollarising safety. Converting injury into money so it can be summed. Keep safety in people and let the max rule handle it.
• Optimistic detection. Using the time a dashboard could alert rather than the time someone would act on it. Measure detection on past incidents or drills.
• Invisible consumers. Missing a downstream system because it was added by another team. Generate the graph from service manifests where possible.
• Scenario stacking. A worst case that requires several independent failures. Split it, or the register fills with unfalsifiable catastrophes.
• Frozen numbers. Bounds computed once at launch and never recomputed after traffic tripled. Recompute on every graph or traffic change.

Trade-offs

Quantified bounds take more effort than a high-medium-low label, and the precision is partly illusory: a fraction estimated from one replay set has wide error. The defence is that every number has a named source and can be challenged, which a label cannot. The max-dominant rule loses information about breadth, since a scenario that is moderate on five dimensions headlines the same as one that is moderate on one; keep the full vector in the record for that reason. Finally, separating impact from likelihood is deliberate. Mixing them in one sitting anchors each estimate on the other, so estimate impact first, hand it to the assessment, and let the likelihood work be done independently.

What to do next

1. Pick your highest-traffic LLM feature and write down its consumer graph, including tools and quiet downstream jobs.
2. Run the blast-radius walk and list every harm node with its reachable fraction.
3. Write three failure scenarios, each naming trigger, fraction, action surface and first detector.
4. Estimate likely and worst detection times from real incidents or a drill, not from dashboard capability.
5. Fill per-unit harm for the five dimensions, with a source beside each number.
6. Compute bounds with the max-dominant rule and record the full dimension vector.
7. For the worst scenario, cost one control that shrinks the window and one that shrinks the fraction, and compare.
8. Commit the graph and scenarios to the repository and recompute them in CI on every change.