Why architecture matters here
Architecture matters here because drift is not an anomaly — it is the equilibrium. Any system where a human can change production faster than a pipeline can will drift, because humans optimise for the incident in front of them and the pipeline is slower than the incident. Treating drift as a discipline problem, solvable by telling people to stop clicking, fails for the same reason it always fails: the click was the correct action at 3am, and the alternative was a longer outage. The architecture must assume drift happens and make its resolution cheap.
The second reason is that undetected drift silently disarms your recovery story. The entire promise of IaC is that if a region vanishes you re-apply the code and get your infrastructure back. That promise holds exactly to the degree that the code describes reality. An estate with six months of unreconciled console changes has code that produces something resembling production, and 'resembling' is discovered during the disaster, when the missing security group rule is between you and the recovery. Drift converts your DR plan into a hypothesis.
Third, drift breaks the review model that justifies IaC's overhead. The reason to accept slow pipelines and pull requests for a firewall change is that every change gets reviewed and recorded. A console change gets neither: no diff, no approver, no rationale, no rollback. If half your changes bypass the pipeline, you are paying IaC's full cost for half its benefit, and the half you lost is the security-relevant half — because the changes people make in a hurry are disproportionately the permission and network ones.
Finally, the classification problem is genuinely hard and genuinely architectural. Deciding whether an ASG's capacity change is expected requires knowing that the field is autoscaled. Deciding whether a new tag is benign requires knowing which tools tag on your behalf. This knowledge lives in your head today; the architecture's job is to move it into declarations — ignore rules, ownership annotations, managed-field markers — so the classifier can act without you. Without that, every drift report needs a human interpreter, and humans stop interpreting when the report is always red.
The architecture: every piece explained
The state file is the participant people forget, and it is why drift detection is a three-way comparison rather than a two-way diff. Declared state is the code. Actual state is what the cloud API returns. The state file is the tool's record of what it believes it created — the mapping from a resource in your code to an ID in the cloud, plus a snapshot of attributes as of the last apply. All three can disagree, and each pairwise disagreement means something different: code versus state means an unapplied change, state versus actual means true drift, and code versus actual with state agreeing with neither means the state file is stale or corrupted.
The refresh is what makes the comparison honest, and it is the expensive part. To know actual state the tool must call the API for every managed resource — thousands of calls, rate-limited, slow. This is why -refresh=false is tempting and why using it converts your plan into a comparison against a memory rather than against reality. A plan with refresh disabled will happily tell you nothing has changed while production has changed substantially. Skipping refresh is skipping the only step that touches the truth.
The classifier is where the design work concentrates. Given a set of differences, it must sort each into a category. Ignorable: fields not yours to manage — an ASG's desired_capacity, a last_modified timestamp, tags applied by a scanner. These need ignore_changes or equivalent so they never appear again; a field that is expected to drift should not be reported as drift, ever. Absorbable: reality is right, the code is stale, and the fix is a pull request bringing the code to match. Hostile: an unauthorised change that must be reverted and investigated. The categories require different actions from different people, which is why a report that does not distinguish them is not actionable.
The reconciliation loop is a product decision, not a setting. Periodic detection — a nightly plan that reports differences — is safe, cheap, and slow to act. Continuous enforcement — a controller that reverts drift within minutes, the GitOps model — closes the window but makes the controller the most dangerous thing in your estate, because a bad merge propagates to production without a human in the loop. Most estates want enforcement for security-relevant resources (IAM, firewall rules, encryption settings) and detection-only for capacity-shaped ones, which means the loop's aggressiveness is a per-resource-class property, not a global mode.
Break-glass is the piece that separates architectures that work from ones that are worked around. People will change production directly during incidents. The choice is whether that path is designed or improvised. A designed path: a role assumable only with a ticket reference, granting console write access for a bounded window, emitting a loud event, and automatically opening a reconciliation task that blocks the next release until resolved. The change still happens, but it happens with a record, a timer, and a forcing function. Without this, every emergency change is indistinguishable from an attack, which means none of them get investigated.
End-to-end flow
Follow a real drift event. It is 02:40 and an on-call engineer is debugging a payment service that cannot reach a new third-party endpoint. The security group blocks egress. The IaC change would take twenty minutes through the pipeline, and the outage is costing money now. They assume the break-glass role, which requires an incident ID, and add the egress rule in the console. The role's session is capped at two hours. A CloudTrail event fires, a bot posts to the incident channel, and a reconciliation ticket opens automatically, linked to the incident. The service recovers.
At 03:00 the nightly drift job runs. It refreshes: for every managed resource it queries the API and compares against the state file. The security group returns an egress rule the state file does not have. That is state-versus-actual disagreement — true drift, unambiguously — and it goes to the classifier.
The classifier checks its rules. The security group's egress rules are not in any ignore list; this is a security-relevant resource, explicitly enforced. It checks the break-glass ledger and finds a session correlated in time and identity with the change. So the classification is not 'hostile' but 'authorised-pending-reconciliation' — the drift is real, the actor is known, the rationale is recorded, and there is already a ticket. The reconciler does not revert it. Reverting would re-break the payment service at 3am to satisfy a policy, which is the reconciler choosing the file over the business.
That restraint is only possible because the ledger exists. Consider the counterfactual: no break-glass role, so the engineer used their standing admin access, and CloudTrail shows a human principal making a console change with no ticket and no session boundary. The classifier has no way to distinguish that from an attacker with stolen credentials doing exactly the same thing — because at the API layer it is exactly the same thing. It must either revert (breaking payments) or page a human (at 3am, to review a change the human on call just made). The break-glass path is what turns an ambiguous event into a classified one, and that is its actual value, ahead of anything it does for compliance.
At 09:00 the ticket surfaces in standup. The engineer writes the pull request adding the egress rule to the code with a comment naming the incident. This is absorption: reality was right, the code was stale, and the resolution moves the code. The PR is reviewed by security — the review the console change never got, now happening while nothing is on fire, which is the better time for it. Merged and applied, the plan is a no-op: the rule already exists, and the apply simply records it in the state file. Drift closed.
Contrast a second event the same night. The classifier finds an IAM role's trust policy widened to allow an external account. No break-glass session correlates. No ticket exists. CloudTrail shows a CI service account that has never touched IAM before. This is hostile, and the response is different in kind: revert immediately (this resource class is enforced), page security rather than the resource owner, and preserve the evidence. The same detection pipeline produced both events; the classifier is what made one a standup item and the other an incident, and that separation is the entire reason the drift report is a thing people read rather than a dashboard that is permanently red.