Cloud FinOps Architecture in Depth

The bill is a data pipeline, not a report

The console cost dashboard is a rendering. The substrate underneath it is a scheduled export you have to switch on: the AWS Cost and Usage Report delivered as Parquet to S3, the Cloud Billing export into a BigQuery dataset, Azure Cost Management exports written to a storage account. FOCUS, the FinOps Open Cost and Usage Specification, is the cross-provider attempt to give those three the same column names. Every serious cost programme queries the export; nobody builds one on screenshots.

The grain is one line item per resource, per usage type, per hour or day. A single virtual machine running for a month is not one row — it is compute hours, attached block storage, provisioned IOPS, snapshot storage and a share of data transfer, each on its own row with its own usage type and operation code. Aggregating that back into “what did this service cost” is the actual work.

The cost columns are not one number either. Unblended is what was charged at the moment of use; blended averages rates across an organisation; amortised spreads a commitment’s upfront payment across its term; net applies negotiated discounts and credits. A team that bought a one-year all-upfront reservation shows an enormous unblended spike in month one and near-zero afterwards, and a flat line on amortised. Both are correct and they answer different questions. Pick one column for engineering reporting — amortised, net of discounts — and one for finance reconciliation, then label every chart with which one it uses. Silent mixing produces arguments that look like allocation disputes and are actually unit-conversion errors.

Granularity and the lag between spend and visibility

The export refreshes several times a day, but the current month is an estimate rather than a fact. Line items are restated as the provider reprocesses usage, and credits, refunds, support fees priced as a percentage of spend, marketplace charges and tax often appear only at close, days after the month ends.

The operational consequence is the one people miss: any control loop that waits for billing truth runs days late. So a cost architecture needs two signals. The slow, authoritative loop is the export, used for allocation, chargeback and forecasting. The fast, approximate loop is telemetry you already collect — running instance-hours from the compute API, node and pod counts, provisioned volume bytes, request counts — multiplied by a rate card that only has to be roughly right. The fast loop pages someone; the slow loop reconciles and settles the argument.

Granularity has a price of its own. Resource-level hourly line items are what make per-service attribution possible, and they multiply row counts into volumes that belong in object storage queried by Athena, BigQuery or Synapse rather than in a reporting database. Some managed services never emit a resource identifier at all, which caps how far attribution can reach regardless of your tagging discipline. Choose granularity per service instead of turning maximum detail on everywhere and then paying to query it.

Tags are the allocation key, and the key is missing

A tag is a key/value pair on a resource. It becomes an allocation key only after a step that surprises almost everyone once: the tag key must be activated for cost allocation in the billing account before it appears as a column in the export, and activation is not retroactive. Spend that occurred before that day carries a blank column permanently. Activate the keys you intend to use on day one of an account, before there is any spend to lose.

Two more mechanics bite early. Tag keys are case-sensitive in the export, so Team, team and TEAM become three columns that each hold a third of your spend. And tags do not propagate the way people assume: an autoscaling group tags the instances it launches only when propagation is explicitly enabled, volumes and snapshots created by a running instance do not inherit the instance’s tags, and resources created by a controller inherit whatever that controller was configured to set, usually nothing.

Enforcement beats convention

There are four enforcement layers and one of them works. A policy check in CI catches infrastructure-as-code but not console clicks. A provider-side tag policy or service control policy that denies creation without the required keys is the only mechanism that prevents untagged spend rather than reporting it; an admission controller does the same job inside Kubernetes. A nightly sweeper mailing owners is cleanup, and cleanup never catches up with creation. Keep the required schema small enough to enforce — owning team, environment, service, cost centre — because four complete keys beat twelve that are each 40% populated.

Then trend unallocated spend as a share of total, weekly. It is the number that decides whether any other cost chart can be believed: if the unallocated share is larger than the differences between the teams you are comparing, every per-team chart is fiction.

-- unallocated share of spend, by week, over a CUR-shaped table
SELECT
  DATE_TRUNC('week', line_item_usage_start_date)      AS wk,
  SUM(line_item_unblended_cost)                       AS total_cost,
  SUM(CASE WHEN COALESCE(resource_tags_user_team, '') = ''
           THEN line_item_unblended_cost ELSE 0 END)  AS untagged_cost,
  100.0 * SUM(CASE WHEN COALESCE(resource_tags_user_team, '') = ''
                   THEN line_item_unblended_cost ELSE 0 END)
        / NULLIF(SUM(line_item_unblended_cost), 0)    AS pct_unallocated
FROM cur
WHERE line_item_line_item_type = 'Usage'
GROUP BY 1
ORDER BY 1;

Where retagging is impractical, backfill at query time with a mapping table — account id to team, resource-id prefix to service, namespace to owner — kept in version control and joined into every report. It is cheaper than a retagging project, it applies retroactively to history no tag can ever reach, and because it is code it can be reviewed when somebody disputes their line.

What tagging cannot reach

A meaningful fraction of a cloud bill is generated by things no team created and no tag describes: NAT gateway data processing, inter-availability-zone transfer, load balancer capacity charges, VPC endpoint hours, KMS request volume, log ingestion and retention, percentage-of-spend support fees, marketplace subscriptions, organisation-level discounts, credits and tax.

Shared platforms make it worse in a more interesting way. One Kubernetes cluster, one Kafka cluster, one warehouse: the nodes carry the platform’s tags, so the export attributes the whole cost to the platform and the tenants inside it are invisible. Recovering per-namespace or per-topic cost means metering inside the platform — splitting node cost across pods by resource requests over time, or warehouse cost by labelled query slot-seconds — and joining that back to billing. That is a second pipeline, and it is the one that makes an internal platform defensible.

There is also a directional trap: transfer and endpoint charges are billed to the account that owns the endpoint, not the one that made the call, so a platform account absorbs cost caused entirely by a consumer’s traffic pattern. Measure the shared bucket before choosing any split. If it is small, absorb it into a platform overhead line and stop — instrumenting it costs more than it explains. If it is large, it is the highest-value thing in the estate to meter properly.

Splitting shared cost, and who each method makes angry

Once the shared bucket is measured it has to be distributed, and every distribution method is a pricing decision with a predictable behavioural consequence.

The politics are not a side effect, they are the mechanism. An even split makes the shared platform look free at the margin, so teams over-consume it and the bucket grows. A precise usage-metered split makes it look expensive at the margin, and the classic outcome is a team building its own thing to escape the internal price — an estate that is cheaper on the report and more expensive in total. You are choosing which distortion you prefer.

Two rules stop the choice destroying trust. Publish the method in enough detail that an engineer can reproduce their own line from the raw export, and freeze it for a defined period, changing it only with notice. A mid-quarter change to the split invalidates every trend chart at once, and a programme that silently rebases its own history loses credibility faster than one that reports an uncomfortable number.

Showback, chargeback, and the account as the real allocation primitive

The strongest allocation key in cloud is not a tag: it is the account, subscription or project boundary. It cannot be forgotten at creation time, it is enforced by the provider rather than by your policy, it maps natively onto the invoice hierarchy, and it carries quotas and blast radius with it. If each team owns its accounts, per-team cost is correct by construction and tagging becomes a refinement rather than a foundation. Multi-tenant accounts are where tagging debt is born.

That makes account topology a FinOps decision as much as a security one, and it belongs in the landing zone design rather than being retrofitted. The tradeoff is real — an account per team per environment multiplies networking, identity plumbing, quota requests and baseline controls — but retrofitting attribution into a shared account is strictly harder.

On top of that boundary, showback reports what a team spent and chargeback moves the money; the behavioural difference is argued in LLM FinOps. What is specific to general cloud spend is which line gets disputed. It is almost never the directly attributed resources — those are visible and obviously the team’s. It is the allocated share of shared and unallocated cost, which the team did not create, cannot see, and cannot reduce by any action available to it. Do not move to chargeback until that share is small, metered, or explicitly absorbed; a first invoice a competent engineer can successfully argue against discredits the whole programme.

Unit economics - the only cost number that survives growth

Absolute spend is a growth metric. It rises when the business is winning and falls when it is shrinking, which makes it useless as a measure of efficiency and dangerous as a target. The number that separates the two is cost per unit of work: per request, per tenant, per order processed, per gigabyte indexed, per monthly active user.

The denominator has to be a metric the business already reports. Reuse the volume number that appears in the product review; a bespoke denominator invented for the cost dashboard invites an argument about the denominator instead of about the cost. The numerator is direct attributed cost plus allocated share of shared cost, recomputed every period from the export rather than carried forward as a fixed rate.

Cost per tenant is the version with the sharpest business consequence, because it turns “our largest customer” into a gross margin and occasionally into “our least profitable customer”. Cost per feature is the version engineering can act on. And there is a clean test for whether the model is real: can you forecast next quarter’s bill by multiplying the business plan’s projected volume by your unit cost? If not, the unit metric is dashboard decoration. If so, cost has become a property of the architecture, and a rising unit cost is an engineering signal in the same way a rising p99 is.

Commitments - coverage and utilisation fail differently

Every provider sells the same trade: give up flexibility, get a lower rate. Reserved instances commit to a family and often a region and size, in standard or convertible forms. Savings plans commit to a level of spend per hour rather than to specific instances, trading a smaller discount for freedom to change shape. Committed use discounts come in resource-based and spend-based variants. The generalisation that holds everywhere is that the deeper and narrower the commitment, the larger the discount — and the discount is what the provider pays you for accepting risk you now own.

That risk is usually realised by your own engineering. A three-year commitment locked to an instance family bets that your architecture will not change for three years, and what most often breaks it is a migration to a newer instance generation, a move to managed or serverless services, or a successful efficiency project: optimisation work stranding the commitment bought to reward the workload it eliminated.

Two metrics describe the book and they fail differently. Utilisation is the fraction of purchased commitment actually consumed; low utilisation means you bought too much, and the waste is invisible in on-demand terms because it never appears as a resource. Coverage is the fraction of eligible on-demand usage a commitment applied to; low coverage means the discount went unclaimed. They are independent. Perfect utilisation with low coverage means you are under-committed and paying list price for most of the fleet; high coverage with mediocre utilisation means the book is badly shaped, committed to the wrong families, regions or hours. A single blended savings figure hides which failure you have.

Buy in tranches. A book bought on one day expires on one day, forcing you to re-commit the whole estate at whatever rates and architecture exist that morning; monthly laddering keeps a portion always maturing and turns a cliff into a rolling decision. Finally, decide how the discount reaches teams: centrally purchased commitments are applied by the provider’s own allocation logic, which will not match your chargeback model, so a team can look cheap purely because discounted hours landed on its instances. Charging a single blended internal rate keeps purchasing variance with whoever made the purchase.

The capacity levers, and which of them compound

Rightsizing recommendations come from utilisation percentiles over a lookback window and have consistent blind spots. Without a guest agent the engine usually sees CPU and not memory pressure. A workload driven by tail latency looks idle at the mean. Capacity provisioned deliberately for failover looks over-provisioned by design. And downsizing an instance covered by a narrow commitment can raise net cost by orphaning it. Treat recommendations as candidates for a human with context, not a queue to approve.

The most reliable finding in any first cost review is not sophisticated: non-production environments running around the clock. Scheduled stop/start for development and test capacity is a mechanical win with essentially no architectural risk.

Elasticity is the larger lever and has its own architecture: matching capacity to demand is covered in cloud autoscaling, and interruptible capacity — pool diversification, drain budgets, checkpointing — in spot capacity. Both are cost levers only to the extent the workload was built to tolerate them, which is why they are engineering projects rather than billing settings.

The distinction worth naming to leadership is between decaying and compounding wins. Rightsizing, schedules, orphan cleanup and commitment purchases are one-off: each is real, each is bounded, and a programme that only chases them exhausts the supply in about two quarters and then looks like it has stopped working. Unit-cost architecture — elasticity, storage tiering, topology that does not generate transfer charges — keeps paying as volume grows. Fund the one-off wins to buy credibility; spend the credibility on the compounding ones.

Storage lifecycle and egress - the lines nobody owns

Storage is the cost line that grows without anyone deciding to grow it. Snapshots from a backup policy with no expiry. Volumes detached from terminated instances and still provisioned. Machine images and their backing snapshots kept in case. Log retention set to never delete. Versioned buckets accumulating noncurrent versions with no expiration rule. Incomplete multipart uploads, invisible in an object listing and charged for anyway. None of these belong to a deploy, so none appear in a team’s mental model of what it spends.

Lifecycle rules are the mechanism — transition objects to colder classes on an age schedule, expire noncurrent versions, abort stale multipart uploads — and the traps are all in the fine print of the storage classes. Transitions are charged per object, so lifecycling a very large number of very small objects can cost more than the storage it saves. Cold and archive tiers carry minimum storage durations, and early deletion is billed as though the object had stayed, so data genuinely read every month does not belong in an archive tier however attractive the per-gigabyte rate. Automatic tiering charges a per-object monitoring fee, which again penalises tiny-object estates.

Egress is the other commonly missed line, and it is missed because it is asymmetric and topological. Inbound is generally free, outbound to the internet is charged per gigabyte, cross-region is charged, and cross-availability-zone traffic is charged — in some services in both directions. What generates it is never a single resource but a shape: a chatty service mesh spread across zones for availability, read replicas in another zone, a pipeline reading in one region and writing in another, a CDN whose hit ratio is low enough that origin fetches dominate. These lines have no owner because a topology decision produced them rather than a team’s workload, which is exactly why they need one.

Anomaly detection and budgets people do not ignore

Cloud spend is seasonal and monotonically growing: weekday and weekend shape, month-end batch shape, release-cycle shape. A fixed threshold on that series fires constantly, is muted within two weeks, and is then absent when it matters. Detection has to compare each allocation key against its own recent baseline, and it has to run on rate of spend rather than cumulative month-to-date — a month-to-date alert fires on the twenty-eighth, when the money is already spent.

Because of the export lag, detection built purely on billing data is structurally days late. Pair it: run a fast detector over inventory and telemetry — instance-hours, node counts, provisioned volume bytes, task counts, NAT bytes — and a slow detector over the finalised export. The fast one is allowed to be approximate because its job is to page; the slow one is authoritative and catches what has no telemetry proxy, such as a rate change or a new marketplace subscription.

What makes an alert actionable is routing and payload. Send it to the owning team, never only to a central finance channel — an alert that lands where nobody can act is a ticket. Include the account and tag, the service and usage type, the delta against baseline, and the specific resource identifiers responsible; “spend is up 40%” with no resource list is closed unread. Annotate planned load tests, migrations and backfills, because a detector that cried wolf during a planned migration is ignored during the real incident a fortnight later. And budget thresholds are a newsletter unless a consequence is attached in advance — a review, a freeze on non-production growth, an approval requirement — agreed with the team that owns the budget.

The operating loop - inform, optimise, operate

The FinOps Foundation’s maturity progression is usually read as a roadmap; it is more useful read as a dependency order. Inform is allocation and visibility: the export, the tags, the shared-cost method, the unit metrics. Optimising before you can attribute produces savings nobody can defend and nobody credits you for. Optimise is levers with named owners, a change window and a tracked outcome — a recommendation without an owner is not a lever. Operate is when it stops being a project: tag enforcement denies at creation, commitments are purchased on a schedule rather than in a panic, cost appears in architecture review beside latency and availability, and unit cost sits on the same dashboard as error rate.

One organisational constraint decides whether any of it works: the person who sees the number must be the person who can change it. A central FinOps function that sees everything and owns nothing produces reports. An engineering team given a budget but no per-service visibility produces resentment and a plausible excuse. The loop closes only when the granularity of attribution matches the boundary of ownership, which is the real reason account topology, tag schema and team structure have to be designed together rather than in sequence.

The team shape that follows is small. A central function owns the pipeline, the definitions, the shared-cost method and the commitment book — the things that must be consistent and that no single team can decide. Engineering owns the levers, because only engineering can pull them. Finance owns the forecast and the reconciliation to the invoice. The central function’s output is not savings; it is a number engineering and finance both believe, and every optimisation the organisation makes is downstream of that.