AI Disaster Recovery + BCP, in depth: asset inventories, immutable signed backups, timeline-based restore points, clean-room recovery and continuity tiers

Inventory the AI assets

Start with an inventory. For each asset record where it lives, who can change it, its recovery time objective (RTO, how long you can be without it), its recovery point objective (RPO, how much change you can lose) and whether it can be rebuilt from something else. Rebuildable assets need their inputs backed up. Assets that cannot be rebuilt need the assets themselves backed up.

Two rows decide most real plans. Evaluation sets are what let you prove a restored system behaves correctly, so losing them makes every other restore unverifiable. And secrets are never restored. Restoring old credentials into a new environment brings back the attacker's access.

Scenarios come from threats

Write scenarios from threats, not from component failures. Each one needs a different restore point and a different continuity tier.

• Destructive or ransomware attack. The attacker encrypts or deletes the model store, the vector database and, if they can reach them, the backups. Recovery depends on a copy the compromised credentials cannot touch.
• Corpus or index poisoning. Malicious documents enter the retrieval corpus and steer answers or carry prompt injections. Backups taken since the injection contain it too. The restore point is the last snapshot before the first bad write, followed by a replay of clean changes.
• Compromised model artifact. A tampered checkpoint, an adapter swapped in the registry, or a pickle-based file that runs code when loaded. Recovery is to a release whose hash matches the signed manifest.
• Training-data poisoning found late. A backdoor is discovered weeks after a fine-tune shipped. The fix is to roll back to the prior release and retrain from cleaned data. RTO is measured in GPU-days, so the continuity tier must carry the business for that long.
• Key or account compromise. A leaked API key or a cloud admin account is abused. Rotate credentials, rebuild from infrastructure as code, and treat every artifact the account could write as suspect.
• Vendor loss. A hosted model is deprecated or the provider is unavailable for days. Not an attack, but it uses the same continuity tiers.

Immutable, isolated, verifiable backups

Backups for AI assets need three security properties. They are immutable: write once, with retention the production account cannot shorten. On AWS this is S3 Object Lock in compliance mode; other clouds have equivalent retention locks. They are isolated: stored in a separate account with separate administrators, so one stolen role cannot delete both copies. And they are verifiable: every backup set carries a manifest of content hashes, signed with a key that never lives in production, for example with Sigstore's cosign.

The manifest is the important part. It turns "we have backups" into "we can prove this file is the one we released".

import hashlib, json, pathlib, time

def sha256(path, chunk=1 << 20):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for block in iter(lambda: f.read(chunk), b""):
            h.update(block)
    return h.hexdigest()

def build_manifest(release_dir, release_id, coupled):
    files = {str(p.relative_to(release_dir)): sha256(p)
             for p in sorted(pathlib.Path(release_dir).rglob("*")) if p.is_file()}
    return {"release": release_id,
            "created": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
            "coupled": coupled,      # e.g. {"embedder": "emb-v3@sha256:...", "index": "idx-2026-10-01"}
            "files": files}

# write manifest.json, then sign it outside production:
#   cosign sign-blob --key vault-key manifest.json --output-signature manifest.sig

The coupled field records versions that must be restored together. A vector index is useless with a different embedding model, because the query vectors no longer live in the same space. Retrieval then fails silently: results come back and are wrong. Store the embedder version, chunking parameters and index build ID as one release bundle, as described in vector DB supply chain.

Choosing the restore point

In a security incident the restore point comes from the attack timeline. Work backwards from the earliest indicator of compromise (IOC) to the last backup that is clean, then add a safety margin, because the first IOC you found is rarely the first event. For corpora and indexes, keep an append-only change log of document writes with author, source and hash. Then you can restore an older snapshot and replay only the writes you trust, instead of losing weeks of legitimate updates.

def choose_restore_point(snapshots, first_ioc, margin_hours=72):
    # snapshots: list of dicts with "taken" (epoch s), "manifest_ok", "id"
    cutoff = first_ioc - margin_hours * 3600
    clean = [s for s in snapshots if s["taken"] < cutoff and s["manifest_ok"]]
    if not clean:
        raise RuntimeError("no verified snapshot predates the incident window")
    return max(clean, key=lambda s: s["taken"])

def replay_writes(change_log, since, until, trusted):
    return [w for w in change_log
            if since <= w["ts"] < until and trusted(w)]   # source allowlist, reviewed authors

This trade-off is the heart of AI DR. A larger margin lowers the chance of restoring the attacker's change and raises the amount of legitimate data you must replay or lose. Make the default margin a written decision before an incident, not a guess during one.

Clean-room restore and the evaluation gate

A clean-room restore rebuilds into an environment the attacker has never touched: new accounts or projects, new credentials, infrastructure built from reviewed code. Then it pulls the chosen release from the vault.

1. Verify the manifest signature, then the hash of every file. Refuse any file not in the manifest.
2. Load weights only from safe formats such as safetensors. Never unpickle a checkpoint during an incident.
3. Rebuild coupled sets together: embedding model, then index from the restored corpus plus trusted replayed writes.
4. Issue fresh secrets and tool credentials. Scope agent tools to the minimum until the postmortem is done.
5. Run the evaluation gate: the golden task set with pass thresholds, plus canary probes for the incident type, such as known trigger phrases for a suspected backdoor or the injected instructions found in poisoned documents.
6. Cut over in stages: shadow traffic, then a small percentage, then everything, with a rollback switch at each step.

The evaluation gate is what makes the result trustworthy. A restore that boots is not a restore that behaves. Keep the golden set and canary probes in the vault with the release, and record the expected pass rates in the manifest, so the gate has a number to compare against.

Continuity tiers

Business continuity asks a different question: while the AI is down or untrusted, how does the business keep working? Define continuity tiers per business process and decide in advance which tier each process falls back to.

Implement tiers as configuration that one on-call person can change, not as code paths written during an incident. Test that each tier works at least once a quarter.

# continuity.yaml, read by the gateway on every request (cached 10 s)
processes:
  customer_support:
    tier: 0
    allowed_tiers: [0, 1, 3, 4]
    fallback_model: support-small-v5      # pre-evaluated, pass rate recorded
  contract_review:
    tier: 0
    allowed_tiers: [0, 4]                 # never degrade to an unvetted model
  internal_search:
    tier: 0
    allowed_tiers: [0, 3]

The allowed_tiers list is a risk decision made by the process owner. A support bot can drop to a smaller model. Contract review should go to people rather than to a model nobody has tested on contracts.

Drills

A plan that has not been exercised is a hypothesis. Run two kinds of drill. A tabletop walks through a scenario with the people involved and finds gaps in ownership and decisions. A restore drill actually rebuilds a release into a clean account from the vault, runs the evaluation gate and measures the time. Record the measured RTO per asset. It is usually longer than the planned one. Index rebuilds and GPU capacity for loading large weights are the common surprises. A restore drill also checks that someone outside the production team can still reach the vault keys. For the response side of the same incidents, see LLM incident response and AI incident runbook design.

Failure modes

• Backups share the blast radius. The production role can delete the backup bucket. Fix with a separate account and compliance-mode retention.
• Restoring the infection. The newest snapshot carries the poisoned documents. Fix with timeline-based restore points and replay of trusted writes.
• Version skew. The index is restored without its embedding model. Retrieval returns confident nonsense. Fix with coupled release bundles.
• Unverifiable restore. Evaluation sets were never backed up, so nobody can show the restored system is good.
• Restored secrets. Old credentials come back with the config and the attacker returns.
• Untested fallback. Tier 1 points to a model nobody has evaluated on the task, so continuity creates its own incident.
• Backup privacy debt. Conversation memory in immutable backups outlives deletion requests. Set retention to match your obligations and document it.

Trade-offs

Immutable retention protects against attackers and also prevents deleting data you are obliged to delete, so keep personal data out of long-locked vaults or encrypt it per tenant so keys can be destroyed. A larger safety margin on restore points is safer and loses more legitimate change. Keeping private copies of base weights costs storage and licence review, and removes the dependency on a vendor still publishing them. More continuity tiers mean more paths to test. Most teams do well with three: fallback model, retrieval-only and human queue.

What to do next

1. Build the asset inventory table for one production AI system, with RTO, RPO and a rebuildable flag per row.
2. Move backups of weights, prompts, configs, corpora and evaluation sets into a separate account with compliance-mode retention.
3. Generate a signed manifest for every release and record coupled versions (embedder, chunker, index).
4. Add an append-only change log for corpus writes so you can restore and replay.
5. Write the restore-point rule and default safety margin down and get it approved.
6. Define continuity tiers per business process in configuration, with pre-evaluated fallbacks.
7. Run one restore drill into a clean account this quarter, gate it with your evaluation set, and record the measured RTO. Review poisoning signals with data poisoning.