AWS CloudTrail, in depth: event types, organization trails, tamper-evident archives and detection

The event record

A CloudTrail event is one JSON object. Log files delivered to S3 are gzipped JSON documents with a top-level Records array. The fields you will query most often are:

• eventTime, eventSource (for example s3.amazonaws.com) and eventName (for example PutBucketPolicy).
• userIdentity: the principal type, ARN, account, access key ID and, for assumed roles, sessionContext with the issuer role and whether MFA was used.
• sourceIPAddress and userAgent. For calls AWS services make on your behalf, the source is a service name, not an IP address.
• requestParameters, responseElements, errorCode and errorMessage. A denied call is still recorded, and patterns of AccessDenied errors are often the first sign of an intruder probing an account.
• eventID (unique per event), readOnly, eventCategory and recipientAccountId.

Two properties of the stream matter for anything you build on it. First, the guide says log files are not an ordered stack trace: events in a file are not in time order, so sort by eventTime yourself. Second, delivery to S3 happens within an average of about five minutes of the call, and AWS states that this is not guaranteed. Treat CloudTrail as an audit record that arrives within minutes, not as a real-time control. Deduplicate on eventID in anything that counts events.

Four event types and what is on by default

CloudTrail logs four event types, and only one is on by default.

Data, network activity and Insights events all cost extra. The first copy of management events delivered to S3 in each Region is free. A second trail that also logs management events is billed for those additional copies, which is why duplicate trails quietly cost money.

The free record every account gets is event history: a searchable, read-only record of the last 90 days of management events in each Region. It needs no setup, but it does not cover data events, it disappears after 90 days, and it is per Region. Use it for quick lookups, not as your audit record.

Trails and organization trails

A trail is a configuration that delivers events to an S3 bucket, with optional delivery to CloudWatch Logs and EventBridge. A multi-Region trail records events in every Region enabled in the account, which is what you want, because attackers like to work in Regions nobody watches. CloudTrail allows five trails per Region, and a multi-Region trail counts as one in each Region.

Global services need care. Since November 2021, events from IAM, STS global endpoint calls and CloudFront are recorded in US East (N. Virginia), us-east-1. A multi-Region trail must include global service events. A single-Region trail outside us-east-1 will miss IAM changes, and those are the changes you most need to see.

In AWS Organizations, an organization trail created from the management account (or a delegated administrator) delivers events from every member account into one bucket, and member accounts cannot delete or change it. Console-created organization trails are multi-Region. One detail from the guide: CloudTrail creates the trail in member accounts even if resource validation fails, for example when a KMS key policy is wrong. A broken organization trail therefore exists and looks fine in a list, while it delivers nothing. Check get-trail-status, not just describe-trails.

# From the management account (or delegated admin), in the home Region
aws cloudtrail create-trail \
  --name org-audit \
  --s3-bucket-name acme-log-archive-cloudtrail \
  --is-multi-region-trail \
  --include-global-service-events \
  --is-organization-trail \
  --enable-log-file-validation \
  --kms-key-id arn:aws:kms:us-east-1:222222222222:key/EXAMPLE-KEY-ID

aws cloudtrail start-logging --name org-audit

# Verify delivery, not just configuration
aws cloudtrail get-trail-status --name org-audit \
  --query '{logging:IsLogging,lastDelivery:LatestDeliveryTime,error:LatestDeliveryError}'

Put the bucket in a dedicated log-archive account that application teams cannot reach. The bucket policy lets the CloudTrail service principal write, scoped to your trail with an aws:SourceArn condition so another account's trail cannot write into your bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {"Sid": "AclCheck", "Effect": "Allow",
     "Principal": {"Service": "cloudtrail.amazonaws.com"},
     "Action": "s3:GetBucketAcl",
     "Resource": "arn:aws:s3:::acme-log-archive-cloudtrail",
     "Condition": {"StringEquals": {"aws:SourceArn":
        "arn:aws:cloudtrail:us-east-1:111111111111:trail/org-audit"}}},
    {"Sid": "Write", "Effect": "Allow",
     "Principal": {"Service": "cloudtrail.amazonaws.com"},
     "Action": "s3:PutObject",
     "Resource": "arn:aws:s3:::acme-log-archive-cloudtrail/AWSLogs/*",
     "Condition": {"StringEquals": {
        "s3:x-amz-acl": "bucket-owner-full-control",
        "aws:SourceArn": "arn:aws:cloudtrail:us-east-1:111111111111:trail/org-audit"}}}
  ]
}

Choosing data events with advanced selectors

Data events are where cost and value both concentrate. Do not switch on every S3 object read in the organization. Use advanced event selectors to choose exactly what to record. The selector fields include eventCategory, resources.type, resources.ARN, readOnly, eventName, eventSource and userIdentity.arn. Operators include Equals, StartsWith and NotStartsWith. This selector records writes and deletes on one sensitive bucket only:

aws cloudtrail put-event-selectors --trail-name org-audit \
  --advanced-event-selectors '[
    {"Name": "Management events",
     "FieldSelectors": [{"Field": "eventCategory", "Equals": ["Management"]}]},
    {"Name": "Writes to the customer-export bucket",
     "FieldSelectors": [
       {"Field": "eventCategory", "Equals": ["Data"]},
       {"Field": "resources.type", "Equals": ["AWS::S3::Object"]},
       {"Field": "readOnly", "Equals": ["false"]},
       {"Field": "resources.ARN", "StartsWith": ["arn:aws:s3:::acme-customer-export/"]}]}
  ]'

Advanced selectors replace any basic selectors on the trail, so keep the management-events selector in the same call. Otherwise the change silently stops management logging.

Proving logs were not altered

An audit log is only useful if you can show it was not edited. With log file integrity validation enabled, CloudTrail hashes every log file with SHA-256. Every hour it delivers a digest file that lists the hashes of the last hour's log files. Each digest is signed with SHA-256 with RSA, using a per-Region key pair, and carries the signature of the previous digest. Together the digests form a chain. Deleting or changing a log file breaks its hash. Deleting or changing a digest breaks the chain.

aws cloudtrail validate-logs --region us-east-1 \
  --trail-arn arn:aws:cloudtrail:us-east-1:111111111111:trail/org-audit \
  --account-id 333333333333 \
  --start-time 2026-10-01T00:00:00Z --end-time 2026-10-02T00:00:00Z

Three limits are easy to miss. validate-logs only checks files that digests reference, and only in the S3 location where CloudTrail delivered them. Logs you copied elsewhere need your own validator. For organization trails it needs --account-id. And validation detects tampering but does not prevent it. Prevention comes from the bucket: S3 Object Lock in compliance mode for your retention period, versioning, a bucket in an account few people can reach, and an SCP that denies cloudtrail:StopLogging, cloudtrail:DeleteTrail and cloudtrail:UpdateTrail to everyone except a break-glass role.

From archive to detection

Archiving is not detection. Two paths turn the trail into alerts. EventBridge receives CloudTrail management events on each Region's default bus with the detail-type AWS API Call via CloudTrail. Rules must therefore exist in every Region you care about, or in a cross-Region forwarding setup. This rule fires when anyone tampers with logging:

{
  "source": ["aws.cloudtrail"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventSource": ["cloudtrail.amazonaws.com"],
    "eventName": ["StopLogging", "DeleteTrail", "UpdateTrail", "PutEventSelectors"]
  }
}

For investigation, query the S3 archive with Athena using a table over the CloudTrail prefix. The CloudTrail console can create one, and partition projection keeps scans cheap. A first query in most incidents asks what a given access key did:

SELECT eventtime, awsregion, eventsource, eventname,
       sourceipaddress, errorcode
FROM cloudtrail_logs
WHERE useridentity.accesskeyid = 'AKIAEXAMPLEKEY'
  AND eventtime BETWEEN '2026-10-01T00:00:00Z' AND '2026-10-03T00:00:00Z'
ORDER BY eventtime;

On CloudTrail Lake: AWS closed it to new customers from May 31, 2026. Existing users keep it, with only critical bug and security fixes, and AWS recommends migrating Lake data to Amazon CloudWatch. For a new design, use the S3 archive with Athena or your SIEM, plus CloudWatch, and do not plan around Lake. GuardDuty also consumes CloudTrail management events itself, so it does not need your trail to be configured.

Worked example: auditing a hot training-data bucket

A team is asked: can we record every read of the ml-training-data bucket? The bucket serves training jobs at a sustained 2,000 GetObject calls per second.

Volume: 2,000 times 86,400 is 172.8 million data events per day, roughly 5.2 billion a month. Assume a record averages about 1.5 KB of JSON. That figure is an assumption; measure your own by sampling a delivered file. On that assumption, the uncompressed volume is about 260 GB per day before gzip. At the per-100,000-event data-event price on the CloudTrail pricing page, 5.2 billion events is a large line item, and the training jobs' own reads generate almost all of it.

Better design: the question behind the request is usually who other than the training jobs read the data. So log reads with a selector that excludes the training role. userIdentity.arn with NotStartsWith on the training role's assumed-role ARN prefix does this. Keep full write and delete logging on the bucket, because writes are rare. The audit question is answered, and the event count drops by orders of magnitude.

Failure modes

• Silent delivery failure. A bucket policy or KMS key policy change stops delivery, but the trail still shows as logging. Alarm on LatestDeliveryError and on the age of LatestDeliveryTime.
• Selector overwrite. A data-event change replaces the selectors and drops management events. Review selector changes as code.
• Region blind spot. A single-Region trail, or alert rules in one Region only, miss activity in Regions nobody uses. Attackers favour exactly those Regions.
• Tampering by an admin. Without an SCP and Object Lock, anyone with cloudtrail:* and s3:* in the account can stop logging and delete evidence.
• Retention shorter than obligations. A lifecycle rule that expires logs at 90 days defeats a one-year or seven-year requirement. Transition logs to colder storage classes instead.
• Double counting. Events can arrive in any order, and pipelines that reprocess files can see the same event twice. Key on eventID.
• Cost spikes. A broad data-event selector on a hot bucket or a busy Lambda can cost more than the workload it audits.

Trade-offs

What to do next

1. Run aws cloudtrail describe-trails and get-trail-status in every account and Region. Confirm one multi-Region organization trail with global service events and validation on, and no duplicate management-event trails.
2. Move the bucket into a log-archive account with versioning, Object Lock and an aws:SourceArn-scoped bucket policy.
3. Attach an SCP denying StopLogging, DeleteTrail, UpdateTrail and PutEventSelectors to everyone except a break-glass role.
4. Add the logging-tamper EventBridge rule in every enabled Region, and an alarm on delivery errors.
5. List your sensitive buckets, tables and functions. Add advanced selectors for writes, and add reads only where a specific question needs them.
6. Create the Athena table, then rehearse one investigation, such as "what did this access key do", before you need it.
7. Schedule a monthly validate-logs run over the previous month and file the output.

Related reading: AWS Config for resource-state history alongside API history, GuardDuty for managed threat detection over the same events, SCPs to protect the trail, S3 Object Lock for the archive, and Athena for queries.