Azure Blob Storage, in depth

Accounts, containers and blob types

A storage account is the unit of naming, billing, redundancy, network rules and keys; its blob endpoint is https://<account>.blob.core.windows.net. Inside it, containers group blobs and carry access and immutability policies. A blob's name can contain slashes, but in the flat namespace a slash is just a character: listing with a prefix and delimiter simulates folders, and renaming a "folder" means copying every blob under it.

There are three blob types, chosen when the blob is created and never changed:

• Block blobs hold almost everything: files, Parquet, images, model checkpoints. They are built from blocks and are the only type that supports access tiers.
• Append blobs only allow appends to the end, which suits logs written by many writers without coordination. Existing content cannot be modified.
• Page blobs are collections of 512-byte pages with random read and write; they back unmanaged VM disks. Managed disks hide them, see Azure Managed Disks.

How block blobs are written

A block blob upload is a two-phase protocol. The client calls Put Block for each chunk, giving each a base64 block ID (all IDs in one blob must be the same length). Staged blocks are stored but invisible. A final Put Block List names the blocks, in order, that make up the blob, and the new version becomes visible atomically with a new ETag. Readers see the old blob or the new one, never a mix. Small blobs can be written in one Put Blob call instead.

The protocol gives you parallel, resumable uploads: blocks upload concurrently, a failed block is simply retried, and a re-run can skip blocks already staged. A blob can have up to 50,000 committed blocks; with current service versions a block can be up to 4,000 MiB, for a maximum blob size of about 190.7 TiB. Uncommitted blocks that are never committed are discarded by the service after about a week. The SDKs do all this for you:

from azure.identity import DefaultAzureCredential
from azure.storage.blob import BlobServiceClient, StandardBlobTier
from azure.core import MatchConditions

svc = BlobServiceClient("https://mlstore01.blob.core.windows.net",
                        credential=DefaultAzureCredential())
blob = svc.get_blob_client("checkpoints", "run-42/step-120000.pt")

with open("step-120000.pt", "rb") as f:
    blob.upload_blob(f, overwrite=True,
                     max_concurrency=8,                   # parallel Put Block
                     standard_blob_tier=StandardBlobTier.COOL)

# Optimistic concurrency: update a manifest only if nobody changed it
mf = svc.get_blob_client("checkpoints", "run-42/manifest.json")
props = mf.get_blob_properties()
mf.upload_blob(b'{"latest": 120000}', overwrite=True,
               etag=props.etag, match_condition=MatchConditions.IfNotModified)

If two writers race, the second If-Match write fails with HTTP 412 and must re-read and retry. For longer exclusive ownership, a lease (15 to 60 seconds, or infinite) locks a blob against writes and deletes by anyone not holding the lease ID.

Access tiers and rehydration

Block blobs live in one of four documented access tiers. Moving down the list, storage gets cheaper and reads, writes and per-GB retrieval get more expensive:

Deleting, overwriting or re-tiering a blob before its minimum period is billed as if it had stayed the full period; a cool blob deleted on day 21 is charged nine more days. Soft-deleted blobs are not counted as deleted until their retention ends. Tier changes between online tiers are immediate. An archived blob cannot be read: you rehydrate it, either with Set Blob Tier (the blob changes tier in place) or by copying it to a new online blob (the archived source stays, avoiding early-deletion fees).

Rehydration has two priorities: standard, which may take up to 15 hours for objects under 10 GB, and high, which may finish in under an hour for objects under 10 GB and costs more. High-priority retrieval is capped at about 10 GiB per hour per account, so bulk restores take far longer than single-blob figures suggest. Completion raises a Microsoft.Storage.BlobTierChanged Event Grid event; subscribe to it rather than polling.

arch = svc.get_blob_client("archive", "2024/q1/events.parquet")
arch.set_standard_blob_tier("Hot", rehydrate_priority="Standard")
print(arch.get_blob_properties().archive_status)   # e.g. rehydrate-pending-to-hot

Lifecycle management

Lifecycle management applies JSON rules once a day to blobs matching a prefix or blob-index tag filter. Rules can tier and delete base blobs, versions and snapshots by age since modification, creation, last access (if access tracking is enabled) or last tier change:

{
  "rules": [{
    "name": "logs-aging",
    "enabled": true,
    "type": "Lifecycle",
    "definition": {
      "filters": { "blobTypes": ["blockBlob"], "prefixMatch": ["logs/"] },
      "actions": {
        "baseBlob": {
          "tierToCool":    { "daysAfterModificationGreaterThan": 30 },
          "tierToArchive": { "daysAfterModificationGreaterThan": 120,
                             "daysAfterLastTierChangeGreaterThan": 7 },
          "delete":        { "daysAfterModificationGreaterThan": 730 }
        },
        "version": { "delete": { "daysAfterCreationGreaterThan": 90 } }
      }
    }
  }]
}

az storage account management-policy create \
  --account-name mlstore01 --resource-group rg-data --policy @policy.json

The daysAfterLastTierChangeGreaterThan condition matters: rehydrating a blob in place does not change its last-modified time, so without it the rule would send a freshly rehydrated blob straight back to archive. Lifecycle rules cannot rehydrate, and new or changed rules can take up to a day to start acting.

Redundancy and data protection

Durability comes from redundancy: LRS keeps three copies in one datacenter, ZRS three copies across availability zones, and GRS or GZRS add an asynchronous copy in the paired region (RA- variants allow reads there). Because geo-replication is asynchronous, check the account's last sync time before a failover; writes after it may be lost.

Redundancy does not protect against your own mistakes. For that, turn on blob soft delete and container soft delete (retention 1 to 365 days), versioning (every overwrite keeps the previous version), and if needed point-in-time restore, which relies on versioning, change feed and soft delete together. For regulatory retention, container or version-level immutability policies (time-based retention and legal holds) make blobs write-once until the policy expires. Versions and snapshots are billed as data, so pair versioning with a lifecycle rule that deletes old versions.

Identity, SAS and network access

Prefer Microsoft Entra ID over account keys. Assign data-plane roles such as Storage Blob Data Reader or Storage Blob Data Contributor at the narrowest scope (a container if possible) to managed identities. Management roles such as Owner do not include data-plane permissions themselves, but Owner and Contributor can list the account keys, and a key gives full control of everything in the account while shared-key authorization is enabled. Once applications no longer use keys, disable shared-key authorization on the account.

When an outside client needs temporary access, issue a user delegation SAS, signed with a key obtained through Entra ID, scoped to one blob or container and a short expiry. It can be revoked by revoking the delegation key, unlike a key-signed SAS, which lives until it expires or the account key is rotated.

from datetime import datetime, timedelta, timezone
from azure.storage.blob import generate_blob_sas, BlobSasPermissions

now = datetime.now(timezone.utc)
udk = svc.get_user_delegation_key(now, now + timedelta(hours=1))
sas = generate_blob_sas("mlstore01", "checkpoints", "run-42/step-120000.pt",
                        user_delegation_key=udk,
                        permission=BlobSasPermissions(read=True),
                        expiry=now + timedelta(minutes=30))

On the network side, private endpoints plus a disabled public endpoint keep traffic on your virtual network; firewall rules are a weaker alternative.

Worked example: tiering checkpoints and logs

A team stores 40 TB of training checkpoints and growing logs. Checkpoints from the last two weeks are read often; older ones are read a few times a year for audits; logs are queried for 30 days then kept for two years for compliance.

• Checkpoints go to Hot on write. A rule moves them to Cold after 21 days, because they are rarely read but must be restorable in minutes when an audit asks. Cold suits this since nobody deletes them within 90 days.
• Logs move to Cool after 30 days and to Archive after 120 days, with daysAfterLastTierChangeGreaterThan guarding rehydrated blobs, and are deleted after 730 days. The account uses GRS, since archive is not available on ZRS or GZRS.
• Deletion safety: blob soft delete for 14 days and versioning with a 90-day version rule, so an accidental overwrite of a checkpoint is recoverable without keeping versions forever.
• Restores: an audit needing 500 GB of archived logs is planned as a standard-priority bulk copy into a separate account, completed over a day or more, not as an emergency high-priority job capped at 10 GiB an hour.

Before deploying, the team prices the plan with the Azure pricing calculator for their region, because storage, transaction, retrieval and early-deletion prices all differ by region and redundancy.

Failure modes

• Surprise early-deletion fees when a job rewrites cool or cold blobs daily. Keep frequently rewritten data in Hot.
• Transaction costs from millions of tiny blobs. Per-operation charges rise in cooler tiers, and archiving many small files makes rehydration slow. Pack small files into larger objects before tiering down.
• Hot partitions. The service partitions by account, container and blob name. Names beginning with a timestamp concentrate new writes in one range; Microsoft's performance guidance suggests spreading names, for example with a short hash prefix, for very high request rates. Watch for HTTP 503 Server Busy and retry with exponential backoff.
• Rehydrated blobs re-archived by a lifecycle rule missing the last-tier-change condition.
• Leaked SAS tokens signed with account keys, valid until expiry. Use user delegation SAS and short lifetimes.
• Archive blocks a redundancy change. Moving to ZRS or GZRS requires rehydrating every archived blob first.

Trade-offs

There is no tier that is cheapest for everything. The right tier is the one that minimises storage plus access plus early-deletion cost for the access pattern you actually have, and that pattern is measurable: enable last-access-time tracking or export storage logs for a few weeks before designing rules for a large account. Likewise, redundancy is a decision about which failure you are buying insurance for, a zone outage or a regional disaster, and the archive tier's restriction to LRS and GRS families means that decision constrains your cost options too.

What to do next

1. Inventory accounts: redundancy, default tier, whether shared-key access and public network access are enabled, and whether soft delete and versioning are on.
2. Write a lifecycle policy for every account holding more than a few TB, including version cleanup and the last-tier-change guard on archive rules.
3. Move applications to managed identities and data-plane roles, then disable shared-key authorization.
4. Test a restore: rehydrate one archived blob with an Event Grid subscription, and undelete one soft-deleted blob, so the runbook is proven.
5. Compare with Azure Files when clients need SMB or NFS, and with Azure Functions for event-driven processing of new blobs.