Model Signing, in depth: per-file manifests, keyless Sigstore signing, the model_signing tool and verifying at the last hop

What a model signature proves, and what it signs

A signature answers two questions: were these exact bytes produced by the party I trust, and have they changed since? It answers nothing else. A signed model can still be backdoored by a poisoned dataset, can still be a pickle that runs code on load, and can still be the wrong version if your policy accepts any version the publisher ever signed. Signing is the integrity and authenticity layer; safe formats, provenance attestations and evaluation sit beside it.

Models are directories, not single files: weight shards, a config, a tokenizer, sometimes Python code for custom architectures. Signing one archive works but forces consumers to download and unpack before verifying, and breaks when hubs serve individual files. The approach taken by model_signing is to hash every file and sign a statement listing them. Per the project README, the signature is a Sigstore bundle stored as JSON containing a DSSE envelope, which wraps an in-toto statement and the signature over it; the statement's subjects are the list of file paths and digests. Conceptually:

# illustrative only: the shape of what gets signed, not the library's exact schema
def build_statement(model_dir):
    subjects = []
    for path in sorted(model_dir.rglob("*")):
        if path.is_file() and not is_ignored(path):      # e.g. git metadata
            subjects.append({
                "name": path.relative_to(model_dir).as_posix(),
                "digest": {"sha256": sha256_file(path)},
            })
    return {"_type": "https://in-toto.io/Statement/v1",
            "subject": subjects,
            "predicateType": "<model signing predicate>",
            "predicate": {}}

envelope = dsse_sign(canonical_json(build_statement(model_dir)), signer)

Listing every file means verification catches a modified shard, a deleted file and an added file, which matters because an attacker who cannot change the weights may still add a Python module that a loader with remote code enabled will import.

Keyless signing with Sigstore

Long-lived signing keys are the classic weakness of code signing: they leak, they are shared across teams, and rotation is painful. Sigstore's keyless flow replaces the key with an identity. The signer generates an ephemeral key pair, proves its identity to the Fulcio certificate authority with an OpenID Connect token, and receives a certificate valid for minutes that binds the public key to that identity. The signature and certificate are recorded in Rekor, an append-only transparency log, and the bundle carries the log's inclusion proof. The private key is then discarded.

A verifier therefore checks three things: the certificate chains to the Sigstore trust root, the signing event is in the transparency log, and the identity and issuer in the certificate match the policy. For a GitHub Actions release workflow the issuer is https://token.actions.githubusercontent.com and the identity is the workflow's URL including its path and ref, which pins signing to one workflow file on one branch. Because the log is public, the publisher can also monitor it for signatures made with their identity that they did not expect, which turns a stolen CI token into a detectable event.

Keyless is not always available: air-gapped environments, or organisations that require hardware-held keys. The tool also supports a private key, a certificate, and PKCS #11 devices, and can point at a private Sigstore instance with a trust configuration file.

Signing and verifying with model_signing

Install the tool from PyPI as model-signing. Signing a local model directory with Sigstore is the default method, and the default signature file is model.sig:

# keyless, interactive or ambient OIDC (e.g. inside GitHub Actions)
model_signing sign bert-base-uncased --signature model.sig

# verify against an expected identity and issuer
model_signing verify bert-base-uncased \
      --signature model.sig \
      --identity "$identity" \
      --identity-provider "$oidc_provider"

# key-based alternative
openssl ecparam -name prime256v1 -genkey -noout -out key.priv
openssl ec -in key.priv -pubout > key.pub
model_signing sign key bert-base-uncased --private-key key.priv
model_signing verify key bert-base-uncased --signature resnet.sig --public-key key.pub

# compute the digest without signing, useful for debugging mismatches
model_signing digest bert-base-uncased

The same operations are available as a Python API, which is what a loader gate should use so the decision is made in the process that loads the weights:

import sys
import model_signing

POLICY = {   # from deployment config, never from the model repository
    "identity": "https://github.com/acme/ml-release/.github/workflows/publish.yml@refs/heads/main",
    "issuer": "https://token.actions.githubusercontent.com",
}

def verify_or_exit(model_dir: str, sig_path: str) -> None:
    try:
        model_signing.verifying.Config().use_sigstore_verifier(
            identity=POLICY["identity"], oidc_issuer=POLICY["issuer"]
        ).verify(model_dir, sig_path)
    except Exception as exc:            # any verification failure is fatal
        sys.exit(f"refusing to load {model_dir}: {exc}")

verify_or_exit("/models/finbert", "/models/finbert.sig")

Keep the signature beside the model directory rather than inside it, so it is never mistaken for model content, and keep the policy in deployment configuration owned by the platform. If the policy lives in the same repository as the model, whoever can swap the weights can also swap the identity they are checked against.

Where verification must run

Where verification runs decides what it protects. Verifying at upload to the registry protects the registry, not the serving node: anyone with write access to the bucket after that check can still swap the files. The robust pattern verifies at the last hop. In Kubernetes, an init container pulls weights and signature into a volume, verifies against the policy, and exits non-zero on failure so the pod never starts; the model server then mounts the verified directory read-only. In a single process, verify and load from the same local path with no writable step in between.

Add verification at three other points as defence in depth: at intake, when a third-party model enters your registry (see the AI supply chain security program for the full gate), in CI before evaluation so you never evaluate one artefact and ship another, and in a periodic sweep of the registry that alerts on anything unsigned. Record the verified manifest digest in your AI bill of materials entry and in the serving logs, so an incident responder can prove which bytes answered a given request.

Container images deserve the same treatment with their own tooling, typically cosign and an admission policy. Signing the model does not sign the server image that loads it, and an attacker who controls the image controls the loader.

Signing also stops at the node boundary. It proves the right files arrived, not that the host which loaded them is the one you think, or that nobody with root on that host altered memory afterwards. When the threat model includes the infrastructure operator, pair signatures with remote attestation of the serving environment, where the measured launch state of a confidential VM or GPU is checked before secrets or weights are released to it; see confidential computing for LLM inference for how clients can verify that chain end to end.

Worked example: catching a swapped shard

A team fine-tunes an open-weight 8-billion-parameter model for internal document classification. Training runs in a GitHub Actions workflow on self-hosted GPU runners and writes safetensors shards, a config and a tokenizer to an object store; a Kubernetes deployment serves it.

1. The release workflow, .github/workflows/publish.yml on main, runs model_signing sign over the output directory with the job's OIDC identity and uploads model.sig next to the directory.
2. The deployment config records the policy: that workflow identity and the GitHub Actions issuer. Nothing else is accepted.
3. The init container verifies the pulled directory. Roughly 16 GB of BF16 weights must be rehashed on every pod start; on fast local NVMe this is seconds to tens of seconds, on network storage it can be minutes, so measure it and cache verified directories by manifest digest on the node rather than skipping the check.
4. Months later an engineer with bucket write access, testing a quantised variant, overwrites one shard in place. The next rollout's init containers fail with a digest mismatch for that file, the pods stay pending, and the alert names the shard. Without signing, the cluster would have quietly served a different model.
5. A second test: someone opens a pull request that changes the workflow and signs from a branch. The certificate's identity carries the branch ref, policy rejects it, and the Rekor entry gives the security team a timestamped record to investigate.

Failure modes and trade-offs

Time-of-check to time-of-use. Verifying a download and then loading from a cache path, or from a hub client that re-fetches, means the verified bytes are not the loaded bytes. Load only from the verified read-only directory.

Files the loader reads but the manifest ignores. If your loader pulls in anything excluded from signing, or fetches extra code at load time, those bytes are unverified. Disable remote code execution in loaders and confirm the manifest covers every file the server opens.

Over-broad policy. Accepting any identity from your GitHub organisation means any workflow in any repository can sign a model you will serve. Pin repository, workflow path and ref. Also decide whether old signatures remain acceptable; a policy that accepts every past release allows rollback to a known-vulnerable model, so pin the expected manifest digest per deployment as well.

Signing unsafe formats. A signed pickle is a pickle you can attribute, not one that is safe. Prefer safetensors and keep scanning, as covered in supply chain attacks on ML.

Trust root and availability. Verification needs current Sigstore trust root material, distributed through TUF; cache it and test what happens when the network is unavailable, so an outage fails closed in a controlled way rather than prompting someone to add a bypass flag. Key-based signing avoids the dependency but brings back key custody and rotation.

Trade-offs: keyless signing removes key management but ties trust to your CI and identity provider, which become the high-value target, so harden runners and branch protection. Per-file manifests verify partial downloads but make verification cost proportional to model size. And signing every intermediate checkpoint is cheap insurance for the lifecycle chain of custody described in AI model lifecycle security, but only the release artefacts need a serving policy.