AWS Secrets Manager Rotation

Core concept

At its heart, rotation solves a compliance and security problem: the longer a credential lives, the larger the window for exposure. Passwords should not be permanent; they should age out and be replaced on a schedule. Secrets Manager rotation automates that replacement.

Built-in rotation is available for AWS-managed services: RDS, Aurora, DocumentDB, and Redshift. Secrets Manager speaks directly to each database, issues a change-password command, and updates the secret. For services outside that list—third-party APIs, internal microservices, OAuth providers—you write a custom Lambda function that knows how to issue a credential update to that system. Secrets Manager invokes your Lambda, passes it the secret, and expects the Lambda to orchestrate the update to the remote issuer.

The result is the same either way: a credential is replaced automatically, and the application continues using the secret name, never knowing the underlying password changed. Secrets Manager handles the versioning.

How it works: the multi-version pattern

Rotation works because Secrets Manager stores multiple versions of a secret and uses staging labels to control which version is current. Every rotation cycle does this:

1. CREATEONLY: Secrets Manager creates a new empty secret version and labels it AWSPENDING.
2. SET: The rotation Lambda reads the AWSPENDING version, issues a new credential from the remote service (database, API, etc.), writes the new credential back to AWSPENDING, and marks it with a ROTATE label.
3. TEST: The Lambda tests the AWSPENDING credential by connecting or calling with it. On failure, the rotation stops; on success, it continues.
4. FINISH: Secrets Manager moves the AWSCURRENT label from the old version to AWSPENDING, making the new credential live. The old version gets labeled AWSPREVIOUS.

The key insight: the application reads the version labeled AWSCURRENT. As long as the new credential works, the application never experiences downtime. The pattern is called multi-user rotation because the old and new credentials can coexist on the remote service during the rotation window, allowing the old client to finish its work while the new client starts.

Built-in rotation for AWS databases

For RDS, Aurora, DocumentDB, and Redshift, AWS provides pre-built rotation handlers that are already aware of each database's password-change API. You enable rotation in the Secrets Manager console (or via CloudFormation), specify a rotation cadence (e.g., every 30 days), and Secrets Manager takes it from there.

Three things happen automatically:

1. Secrets Manager creates an AWS Lambda execution role with permissions to call the database's password-change operation.
2. Secrets Manager invokes a pre-written Lambda function you do not see or manage (it lives in AWS's own account).
3. That Lambda executes the four-step rotation cycle above, issuing ALTER USER commands to the database.

You do not write or maintain the Lambda; AWS does. You just turn on the feature, and it works. The trade-off: built-in rotation is limited to the databases AWS supports. For everything else—RabbitMQ, Postgres on EC2, a proprietary API—you write custom rotation.

Custom Lambda-based rotation

To rotate a credential for a service AWS does not natively support, you write a Lambda function that Secrets Manager invokes at rotation time. Your function receives three arguments: service_client (a Boto3 Secrets Manager client), secret_id (the ARN or name of the secret), and token (a UUID Secrets Manager generates for this rotation run).

Your Lambda implements the four steps:

1. CREATEONLY: Retrieve the AWSPENDING version (which Secrets Manager creates automatically) or skip if it already exists.
2. SET: Generate or request a new credential from the remote issuer. For a REST API, call the key-rotation endpoint. For a database on EC2, connect and run the password-change command. Write the new credential back to Secrets Manager using put_secret_value().
3. TEST: Validate the new credential by connecting or calling the remote service. If validation fails, raise an exception and Secrets Manager rolls back—it deletes AWSPENDING and does not move the AWSCURRENT label.
4. FINISH: Call update_secret_version_stage() to move AWSCURRENT from the old version to the new one.

Because you control the Lambda, you control the rotation flow. You can add custom pre-checks (e.g., 'verify this cluster is not under high load'), custom logging, custom error handling. The cost is that you own the testing, monitoring, and correctness of the rotation logic.

Rotation lifecycle and versioning

Every secret has a version ID (a UUID) and a set of staging labels that attach versions to roles in the rotation process. The lifecycle looks like this:

When you rotate manually (via the console or API), or when a scheduled rotation fires, Secrets Manager walks the CREATEONLY→SET→TEST→FINISH cycle, managing the labels. At any point in time, at most one version has the AWSCURRENT label; applications read that label, not a version ID, so they always get the live credential.

The AWSPREVIOUS label is kept for a short window (by default, none; you can request it) so that if a rotation fails catastrophically and you need to roll back manually, the old credential is still available in Secrets Manager. After the rollback period expires, Secrets Manager cleans up old versions automatically.

IAM permissions and rotation roles

For a rotation to succeed, the Lambda function (or the AWS-managed rotation handler) must have IAM permissions to:

1. Read the secret from Secrets Manager.
2. Update the secret version and move staging labels.
3. Call the credential-issuing service (RDS, DocumentDB, an API, etc.).

AWS creates and manages the role for built-in rotation automatically. For custom Lambda, you create a role with policies like:

• secretsmanager:GetSecretValue — read the secret
• secretsmanager:UpdateSecret — write new versions
• secretsmanager:DescribeSecret — check secret metadata
• Service-specific permissions: rds:ModifyDBInstance for RDS, ec2:DescribeSecurityGroups for network access, etc.

The rotation Lambda must also live in a VPC (or be able to reach one via VPC endpoint) if the credential-issuing service is in a private network. This often trips teams: a Lambda that cannot reach the database because it is running in the default VPC or a different subnet will fail silently every rotation. Test the network path before enabling automated rotation.

Rotation cadence and scheduling

When you enable rotation, you specify a schedule: rotate every 30 days, every 7 days, every day. Secrets Manager creates a CloudWatch Events rule that fires at the interval you specify. On each firing, Secrets Manager invokes the rotation Lambda and orchestrates the four-step cycle.

The schedule is not wall-clock time (e.g., every Monday at 9 AM); it is duration-based (every 30 days from the last rotation). If a rotation fails, the next attempt waits 30 days from the start of the current rotation, not the last successful one. In practice, this means if rotation breaks and you do not fix it, the next attempt fires up to 60 days after the previous success (30 days of no rotations, then a new one starts). For mission-critical credentials, monitor rotation success in CloudWatch and alert on failures.

You can also rotate manually at any time by calling rotate_secret() in the SDK, or by clicking the console. Manual rotations do not reset the schedule; the next automatic rotation still fires at the regular interval.

Handling rotation failures

When rotation fails—the Lambda throws an exception, the credential test fails, the remote service rejects the update—Secrets Manager does not move the AWSCURRENT label. The old credential stays live, and applications keep working. The rotation cycle is marked failed in the audit trail and in CloudWatch Events, but the production credential is never corrupted.

Common failure modes:

1. Network unreachable. The Lambda cannot reach the credential-issuing service. Check security groups, VPC configuration, and route tables.
2. Permission denied. The database user, API key, or service account performing the rotation does not have permission to create new credentials. Verify IAM roles and database privileges.
3. Validation failed. The new credential was created but does not work when tested. Secrets Manager rolls back and keeps the old credential current.
4. Timeout. The Lambda takes longer than the CloudWatch Events timeout (typically 60 seconds for the entire rotation). Complex rotations need longer timeouts; consider a step-function wrapper if rotation takes minutes.

When rotation fails, fix the underlying issue (network, permissions, Lambda code), and either wait for the next scheduled rotation or trigger one manually. Secrets Manager does not auto-retry within a cycle; it waits for the next scheduled window to try again.

Monitoring and logging rotation events

Rotation is a background process, and teams often miss failures because they don't watch for them. Three tools surface rotation health:

CloudWatch Events / EventBridge: Every rotation start and end fires an event. You can log these events to CloudWatch Logs or send them to SNS, Lambda, or a SIEM. A rule like 'SecretRotationEvent with status=Failed' sends an alert when rotation breaks.

CloudTrail: Every Secrets Manager API call (rotation or manual) is logged in CloudTrail. Query UpdateSecretVersionStage to audit who changed which labels when. This is your audit trail for compliance.

CloudWatch Logs (custom): Your rotation Lambda can write to CloudWatch Logs. Include detailed logs: what credential was generated, what test passed or failed, what labels were moved. When rotation breaks at 3 AM, the logs are your first clue.

The pattern that works is: fire CloudWatch Events to CloudWatch Logs, set up a metric filter to count failures, create an alarm on that metric, and escalate to your on-call team. Silent rotation failures are the worst; visible failures you can fix.

Application integration during rotation

Applications must be rotation-aware without complex logic. The pattern is simple: always read the AWSCURRENT version. Most AWS SDK clients do this automatically if you tell them the secret is managed by Secrets Manager.

For example, in Python with Boto3:

import boto3
import json

client = boto3.client('secretsmanager')
response = client.get_secret_value(SecretId='my-db-secret')
secret = json.loads(response['SecretString'])
# now use secret['username'] and secret['password']
# Secrets Manager always returns the AWSCURRENT version

The application does not care about versioning or labels; Secrets Manager's get_secret_value() always returns the AWSCURRENT version. If rotation happens mid-request, the application will read either the old or the new credential (whichever is CURRENT at the moment of the call), so there is no downtime.

For clients that cache credentials (common in connection pools), rotation works because the next time they call get_secret_value(), they get the new one. Connection pools with long-lived connections to the old credential will eventually fail when the old password is revoked on the remote side; well-designed pools handle this by closing and re-opening connections on auth errors. Test this behavior before deploying rotation to production.

Trade-offs and gotchas

Tight coupling to rotation Lambda. Custom rotation Lambda bugs are production bugs—they break all future rotations. Test the Lambda in a lower environment thoroughly, including failure cases (what happens if the API times out? if the credential is already taken?).

Rotation window timing. If rotation takes 10 seconds but you have 1,000 requests per second on that credential, 10,000 requests might read the old credential while the new one is being set. This is fine, but applications must tolerate it. Do not assume the old credential is gone the instant rotation is called.

Revocation delay. Secrets Manager can rotate the password in its store instantly, but revoking the old credential on the remote service (e.g., dropping the old DB user) is often manual or asynchronous. Until the old credential is actually revoked, both credentials work, and an attacker with the old one still has access. Rotation is about changing credentials, not instantly eliminating old ones.

Lambda cold starts and timeouts. A Lambda rotation that times out fails silently. If your rotation Lambda has cold-start latency or complex logic, increase the timeout or use provisioned concurrency to keep the function warm.

Best practices

Rotate only what matters. Not every secret needs automatic rotation. A service account password used by a batch job that changes users yearly: rotate it. A read-only API key from a third party: maybe not. Rotation adds operational complexity; use it where the compliance or security benefit outweighs that cost.

Start with a long cadence. Do not rotate every day. Start with every 30 days, observe rotation in a lower environment, confirm that applications handle it gracefully, and only then move to production. Shorten the cadence if compliance requires it.

Test custom rotation thoroughly. Simulate the SET step (new credential created), the TEST step (new credential validated), and the FINISH step (new credential is current) in an isolated environment. Test failure modes: what happens if the remote service rejects the update? if the test fails? if the Lambda times out?

Monitor rotation success. Set up a CloudWatch alarm on rotation failures. When rotation breaks, you want to know within hours, not weeks. Log every rotation event and test those logs in a lower environment to confirm they are readable and actionable.

Document the rotation process. If you write a custom Lambda, document what credentials are rotated, how they are issued, who can revoke them, and what to do if a rotation fails catastrophically. Future you (or your on-call team) will thank you at 3 AM.