Amazon Timestream, in depth: LiveAnalytics internals, late data and query cost, and the move to Timestream for InfluxDB

One name, two engines

The diagram is the summary. LiveAnalytics is serverless: you never choose instances, and you pay for writes, for storage in each tier, and for query compute. InfluxDB is instance-based: you choose an instance size and storage, and you get InfluxDB's own APIs, line protocol and tooling. The InfluxDB 3 option comes in two versions. Core is limited to a single node and has no compaction, so AWS positions it for recent data, typically a few days. Enterprise supports multi-node clusters, compaction, and query-only nodes that scale reads separately from ingest.

The LiveAnalytics data model and storage tiers

A LiveAnalytics table holds records. Each record has a timestamp, a set of dimensions (string name-value pairs that identify the series, such as site=plant-7 and device_id=d-0042), a measure name, and one or more measure values. A table can have up to 128 dimensions, and dimension values are always strings. A multi-measure record carries up to 256 measures in one record, which is how you should model a device that reports temperature, humidity and voltage together; single-measure records multiply both the record count and the write cost.

Data lands in the memory store, which is optimised for fast point and recent-range queries, and moves to the magnetic store when it is older than the table's memory retention. Memory retention can be set from 1 hour to 8,766 hours (one year) and defaults to 6 hours; magnetic retention runs from 1 day to 73,000 days (200 years). Queries see both tiers as one table. Writes are accepted up to 15 minutes in the future and, by default, only within the memory-store window; anything older is rejected unless you enable magnetic store writes.

Writing efficiently

Writes go through WriteRecords, which accepts at most 100 records per call and up to 2 KB per record. Put shared dimensions in CommonAttributes so each record carries only what differs, batch to 100, and handle RejectedRecordsException, which reports per-record reasons such as a timestamp outside the retention window or a duplicate with a different value.

import time, boto3
ts = boto3.client("timestream-write")

def flush(batch, site):
    common = {"Dimensions": [{"Name": "site", "Value": site}],
              "MeasureName": "reading", "MeasureValueType": "MULTI",
              "TimeUnit": "MILLISECONDS"}
    records = [{
        "Dimensions": [{"Name": "device_id", "Value": r["device"]}],
        "Time": str(r["ts_ms"]),
        "MeasureValues": [
            {"Name": "temperature", "Value": str(r["temp"]), "Type": "DOUBLE"},
            {"Name": "humidity",    "Value": str(r["hum"]),  "Type": "DOUBLE"},
            {"Name": "voltage",     "Value": str(r["volt"]), "Type": "DOUBLE"},
        ],
    } for r in batch]
    for i in range(0, len(records), 100):            # hard limit: 100 per call
        try:
            ts.write_records(DatabaseName="iot", TableName="readings",
                             CommonAttributes=common, Records=records[i:i + 100])
        except ts.exceptions.RejectedRecordsException as e:
            for rej in e.response["RejectedRecords"]:
                dead_letter(records[i + rej["RecordIndex"]], rej["Reason"])

Producers that already publish to a stream usually write through a consumer rather than directly, which buffers bursts and gives you replay; Kinesis Data Streams is the common choice. Keep the consumer's batch size at a multiple of 100 and its retries idempotent: Timestream treats a record with the same dimensions, measure name and timestamp as the same point.

Late data, magnetic writes and partition keys

Late data is the most common LiveAnalytics surprise. A gateway that was offline for a day uploads its buffer, every record is older than the 6-hour memory window, and every record is rejected. Two fixes exist. You can lengthen memory retention, which costs more because memory storage is the expensive tier. Or you can enable magnetic store writes, which accept late records directly into the magnetic tier asynchronously and can send records that still fail to an S3 location for inspection.

ts.update_table(
    DatabaseName="iot", TableName="readings",
    RetentionProperties={"MemoryStoreRetentionPeriodInHours": 24,
                         "MagneticStoreRetentionPeriodInDays": 730},
    MagneticStoreWriteProperties={
        "EnableMagneticStoreWrites": True,
        "MagneticStoreRejectedDataLocation": {"S3Configuration": {
            "BucketName": "iot-ts-rejects", "ObjectKeyPrefix": "readings/"}},
    },
)

Magnetic writes have their own limit to plan around: each database allows 250 active magnetic store partitions, and a partition stays active for up to six hours after it receives data. A large historical backfill that touches many time ranges and series at once can hit that ceiling and be throttled; spread backfills over time, or use the batch load feature for bulk history from S3.

Tables can also use a customer-defined partition key: a dimension you name as the partitioning key through CompositePartitionKey, with EnforcementInRecord set to REQUIRED or OPTIONAL. Pick the dimension most of your queries filter on, such as a fleet or tenant identifier, so queries for one value touch less data.

Queries and what they cost

Queries use SQL with time-series functions. ago() gives relative windows, bin() buckets timestamps, and multi-measure values appear as ordinary columns. Always bound the time range and filter on dimensions; the engine prunes by time and by partition key, and an unbounded query scans the whole history.

q = boto3.client("timestream-query")
sql = """
SELECT device_id, bin(time, 5m) AS bucket,
       avg(temperature) AS avg_temp, max(voltage) AS max_volt
FROM "iot"."readings"
WHERE time > ago(2h) AND site = 'plant-7'
GROUP BY device_id, bin(time, 5m)
ORDER BY bucket DESC
"""
for page in q.get_paginator("query").paginate(QueryString=sql):
    for row in page["Rows"]:
        handle(row["Data"])

Query compute is measured in Timestream Compute Units. One TCU is 4 vCPUs and 16 GB of memory, billed per second with a 30-second minimum. The account setting MaxQueryTCU caps how many TCUs queries may use at once; it defaults to 200 and can be set from 4 to 1,000 in multiples of 4. Lowering it is a cost guardrail: queries queue or slow down instead of running up the bill. Dashboards that refresh every few seconds are the classic cost leak, because each refresh is a query. Pre-aggregate them with scheduled queries that write rollups, such as 5-minute averages per device, into a smaller table, and point the dashboard there.

Timestream for InfluxDB

For new workloads, Timestream for InfluxDB gives you InfluxDB with AWS handling provisioning, patching and backups. You choose the engine version and the instance and storage size, and you talk to it with InfluxDB clients, Telegraf and line protocol rather than AWS SDK write calls. With InfluxDB 3 you write line protocol and query in SQL or InfluxQL:

# pip install influxdb3-python
from influxdb_client_3 import InfluxDBClient3

client = InfluxDBClient3(host="https://<your-instance-endpoint>", token=TOKEN, database="iot")

# measurement,tag_set field_set timestamp
client.write(record="readings,site=plant-7,device_id=d-0042 "
                    "temperature=21.4,humidity=0.41,voltage=3.31 1759550400000000000")

table = client.query(
    query="""SELECT device_id, date_bin(INTERVAL '5 minutes', time) AS bucket,
                    avg(temperature) AS avg_temp
             FROM readings
             WHERE time > now() - INTERVAL '2 hours' AND site = 'plant-7'
             GROUP BY device_id, bucket""",
    language="sql")

The mapping from LiveAnalytics is direct: dimensions become tags, measures become fields, the measure name becomes the measurement. What changes is responsibility. Capacity is now an instance you size and watch, series cardinality (the number of distinct tag combinations) drives memory and index cost, and with InfluxDB 3 Core you must keep the working set to recent data because there is no compaction. The practical differences between InfluxDB and the alternatives are covered in TimescaleDB versus InfluxDB.

Worked example: a 20,000-sensor fleet

Worked example. 20,000 sensors each report three values every 10 seconds, and the business wants 2 hours of fast dashboards and 2 years of history.

On LiveAnalytics, multi-measure records give 2,000 records per second, which is 20 WriteRecords calls per second at 100 records each. Single-measure records would be 6,000 records and 60 calls per second for the same data. Memory retention of 24 hours covers the dashboards and a day of gateway outages; magnetic writes catch anything later. A scheduled query rolls up to 5-minute averages, which is 20,000 x 288 = 5.76 million rows per day instead of 172.8 million raw records, and the dashboard reads only the rollup.

On InfluxDB, the same feed is 2,000 lines per second carrying 6,000 field values, with 20,000 series if site and device_id are the only tags. That is modest cardinality. The danger is adding a tag such as a firmware build string or a request ID, which multiplies series and memory. Two years of raw 10-second data on Core is outside its design point, so either downsample into a long-term store or choose Enterprise.

Failure modes

Failure modes, with the signal that reveals each:

• Rejected late data. RejectedRecordsException reasons mentioning retention. Lengthen memory retention or enable magnetic writes with an S3 rejects location.
• Clock skew on devices. Records more than 15 minutes in the future are rejected; a device with a bad clock loses all its data. Stamp time at a trusted gateway if devices drift.
• Single-measure explosion. Write costs several times higher than expected. Move to multi-measure records.
• Backfill throttling. Hitting 250 active magnetic partitions. Slow the backfill or use batch load.
• Query cost spikes. TCU usage rising with dashboard refreshes. Lower MaxQueryTCU, bound time ranges, and serve dashboards from scheduled-query rollups.
• InfluxDB cardinality blow-up. Memory pressure after a new tag ships. Review tag design in code review the way you review database indexes.
• Planning a new LiveAnalytics account. It cannot be created. Design on InfluxDB or another store from the start.

Choosing and migrating

If you already run LiveAnalytics and it meets your needs, there is no forced migration; AWS states existing workloads are unaffected. Tighten its cost with multi-measure records, a sensible memory window, a TCU cap and rollups. Plan a migration if you need new accounts outside your existing payer account, or if you want the InfluxDB ecosystem. For new work, Timestream for InfluxDB fits device and infrastructure telemetry with real-time dashboards. If your time-series data is mostly operational metrics for AWS resources, CloudWatch may already hold it. If the data is really wide-row events queried by key rather than by time window, a key-value store such as Amazon Keyspaces may fit better.

A migration off LiveAnalytics has three steps. First, dual-write: send new data to both engines from the same consumer so the target fills while the source keeps serving. Second, export history with the UNLOAD statement, which writes query results to S3, one time range at a time, and convert the rows to line protocol (dimensions to tags, measures to fields). Third, rebuild each dashboard query in the target's SQL or InfluxQL, compare results for the same window on both engines, and only then switch reads.

What to do next

1. Find out which Timestream engine each of your accounts can use; new accounts cannot create LiveAnalytics tables.
2. On LiveAnalytics, convert single-measure writes to multi-measure records with CommonAttributes.
3. Measure how late your latest data arrives and set memory retention or enable magnetic writes accordingly.
4. Set MaxQueryTCU deliberately and move dashboards to scheduled-query rollups.
5. Add a dead-letter path for rejected records and alert on its rate.
6. For new workloads, prototype on Timestream for InfluxDB, list your tags, and compute series cardinality before you load real data.
7. Choose InfluxDB 3 Core only if you keep recent data; otherwise plan Enterprise or downsampling.