AI Safety Frameworks, in depth: NIST AI RMF, ISO/IEC 42001, the EU AI Act and frontier-lab policies, crosswalked into controls, eval gates and evidence an engineering team can run

Four kinds of framework

The word framework covers documents with very different authority and purpose. Separating them avoids the most common confusion, which is treating a voluntary guide as a legal requirement or a lab's internal policy as a standard you can adopt.

• Risk management frameworks tell you how to organize the work. The NIST AI Risk Management Framework is the main example: voluntary, outcome-oriented, not certifiable.
• Management-system standards define auditable processes. ISO/IEC 42001:2023 specifies an AI management system that an organization can be certified against, in the same family style as ISO/IEC 27001 for information security.
• Regulation creates legal obligations. The EU AI Act assigns duties by risk tier and by role (provider, deployer, importer), with penalties.
• Frontier capability policies are published by model developers about their own models: they define dangerous-capability thresholds, the evaluations that detect them, and the safeguards required before training further or deploying.

Alongside these sit threat catalogues such as the OWASP Top 10 for LLM applications and MITRE ATLAS. They are not frameworks for governance, but they are the best source of concrete risks to put in a register.

NIST AI RMF and the generative AI profile

NIST published AI RMF 1.0 in January 2023. It is organized around four functions. Govern sets up policies, roles, accountability and culture, and applies across the others. Map establishes context: the intended use, the people affected, the system's components and its known limits. Measure assesses the mapped risks with quantitative and qualitative methods, including testing, evaluation, verification and validation. Manage prioritizes risks and acts on them: mitigate, transfer, accept or avoid, then monitor and respond to incidents.

The framework describes trustworthy AI through characteristics: valid and reliable; safe; secure and resilient; accountable and transparent; explainable and interpretable; privacy-enhanced; and fair with harmful bias managed. In July 2024 NIST published AI 600-1, a generative AI profile that applies the framework to generative models and lists risks specific or amplified by them, such as confabulation, information integrity, information security, harmful bias, privacy, intellectual property, and dangerous or violent content, each with suggested actions keyed to the four functions. NIST continues to publish companion material, so check nist.gov for revisions before citing the profile in a formal document.

For engineers the useful part is the Map and Measure functions: they are a checklist for describing a system precisely and for choosing what to evaluate.

ISO/IEC 42001

ISO/IEC 42001:2023 is a management-system standard. It asks the organization to define the scope of its AI management system, set an AI policy and objectives, assess AI risks and the impacts of AI systems on individuals and society, select controls from its annex or justify excluding them, operate them, monitor, audit internally, review, and improve. Its value is that it is certifiable: a customer or regulator can rely on an external audit rather than on your own description.

Its cost is that it is about process. A certificate says the organization runs a management system; it does not say any particular model is safe. Treat it as the container your engineering controls live in, not as a substitute for them.

The EU AI Act and its timetable

The AI Act entered into force on 1 August 2024 and applies in stages. Prohibited practices, such as certain manipulative systems and social scoring, have applied since 2 February 2025. Obligations for providers of general-purpose AI models have applied since 2 August 2025, with extra duties for models presumed to carry systemic risk, a presumption tied to training compute above 10^25 floating-point operations. Transparency duties under Article 50, including telling people they are interacting with an AI system and marking synthetic content, apply from 2 August 2026.

High-risk systems are the heaviest tier: risk management, data governance, technical documentation, logging, human oversight, accuracy and robustness, and conformity assessment. In 2026 the EU adopted a Digital Omnibus amendment that moved the dates: stand-alone high-risk systems listed in Annex III, such as those used in employment, credit scoring, education and critical infrastructure, now have until 2 December 2027, and AI in products covered by existing EU product-safety law under Annex I until 2 August 2028. Timetables have already moved once, so track the official text rather than a blog summary.

Most LLM applications built by companies are not high-risk systems, but many are deployers of general-purpose models and are subject to the transparency duties. The first engineering task is classification: record for each system which tier and which role applies, and why.

Frontier-lab capability policies

Anthropic's Responsible Scaling Policy, OpenAI's Preparedness Framework and Google DeepMind's Frontier Safety Framework share a structure. Each defines capability thresholds in a few high-consequence domains, such as biological and chemical weapons uplift, offensive cyber capability, and autonomous AI research; runs evaluations before and during deployment to test whether a model is approaching a threshold; and commits to safeguards, deployment restrictions or pauses when it does. OpenAI's version 2, published in April 2025, uses High and Critical thresholds across three tracked categories: biological and chemical, cybersecurity, and AI self-improvement. DeepMind's framework is built around Critical Capability Levels. Anthropic rewrote its policy as version 3.0 in February 2026, adding published risk reports and roadmaps, and has revised it several times since.

All three change often, so read the current version rather than a summary. For most application teams these policies are not obligations; they are context about the models you build on and a template for your own thinking. The evaluation discipline behind them is explained in the dangerous capability evaluations guide.

The crosswalk

Mapping activities to frameworks shows why one engineering system can serve all of them.

Operationalizing: the engineering system

Build five artifacts, each as code or data in version control, not as slides.

1. Inventory. One record per AI system: purpose, users, model and version, data sources, tools it can call, deployment surface, owner, and its classification under each applicable framework.
2. Risk register. One entry per risk per system, with likelihood, severity, owner, the controls that treat it and the residual risk accepted by a named person.
3. Control library. Each control defined once, with the framework clauses it satisfies, how it is tested and where its evidence lives.
4. Release gates. Evaluation suites whose results are compared to thresholds in CI; a failed gate blocks deployment unless a recorded exception exists.
5. Evidence store. Versioned evaluation results, approvals and incident records, retrievable per release.

A register entry and a gate that reads it can be this simple:

# risks/support-agent.yaml
system: support-agent
risks:
  - id: R-07
    title: Agent issues refunds outside policy after prompt injection
    severity: high
    owner: payments-eng
    controls: [C-tool-allowlist, C-refund-approval, C-injection-evals]
    gates:
      - suite: injection_refund_v3
        metric: attack_success_rate
        max: 0.01
      - suite: refund_policy_regression
        metric: pass_rate
        min: 0.98

import json, sys, yaml

def gate(register_path, results_path):
    reg = yaml.safe_load(open(register_path))
    results = json.load(open(results_path))   # {suite: {metric: value}}
    failures = []
    for risk in reg["risks"]:
        for g in risk.get("gates", []):
            value = results.get(g["suite"], {}).get(g["metric"])
            if value is None:
                failures.append(f"{risk['id']}: no result for {g['suite']}")
            elif "max" in g and value > g["max"]:
                failures.append(f"{risk['id']}: {g['metric']}={value} > {g['max']}")
            elif "min" in g and value < g["min"]:
                failures.append(f"{risk['id']}: {g['metric']}={value} < {g['min']}")
    for f in failures:
        print("GATE FAIL", f)
    return 1 if failures else 0

if __name__ == "__main__":
    sys.exit(gate(sys.argv[1], sys.argv[2]))

A missing result fails the gate, deliberately: an evaluation that silently stopped running is the most common way these systems decay. The suites themselves come from an evaluation pipeline and from red-team findings converted into regression tests.

Worked example: a support agent with a refund tool

A retailer deploys an LLM agent that answers order questions and can issue refunds up to a limit. Map: the inventory records the model, the order and refund tools, customers as users, and order data as a source containing personal data. Classification: not an Annex III high-risk use; subject to the Article 50 duty to tell users they are talking to an AI system; the retailer is a deployer of a general-purpose model. Measure: the register lists prompt injection leading to unauthorized refunds, leakage of other customers' data, and confabulated policy statements, each with an evaluation suite. Manage: controls include a tool allowlist, human approval for refunds above a threshold as described in the human-in-the-loop guide, output filtering for personal data, and a disclosure banner. Govern: payments engineering owns the refund risk, and the head of support signs the residual risk.

A model upgrade raises the injection suite's attack success rate from 0.4 percent to 2.1 percent. The gate fails and the release stops. The team tightens the tool-call policy, reruns the suite at 0.6 percent, and ships. The evidence store now holds both runs and the change, which is precisely what an ISO auditor or a customer questionnaire asks for.

Failure modes

• Paper compliance. Policies exist but no gate can block a release. If nothing has ever been stopped, the system is not running.
• Framework shopping. Adopting the lightest framework for each audience. Maintain one control library and map it to all.
• Stale evaluations. Suites that no longer reflect current attacks pass forever. Refresh them from red-team findings and incidents.
• Scope drift. A new tool or data source is added without re-running Map. Make inventory changes part of code review.
• Owner-less risks. A risk without a named owner is accepted by default. Require an owner to merge the register entry.
• Over-claiming. A 42001 certificate or an RMF alignment statement does not make a model safe; say what was assessed.

What to do next

1. Inventory every AI system in production and record its purpose, model, tools, data and owner.
2. Classify each system under the EU AI Act tiers and roles, and note which transparency duties apply from 2 August 2026.
3. Seed a risk register from the NIST generative AI profile, the OWASP LLM Top 10 and your own incidents, with an owner per risk.
4. Define controls once and map each to NIST functions, ISO/IEC 42001 clauses and AI Act articles.
5. Wire evaluation suites into CI with thresholds, and fail on missing results.
6. Store evidence per release, and review the register whenever a model, tool or data source changes. For the runtime layer, see defense in depth for LLM systems.