Why architecture matters here

Paimon fails on wrong merge engine + missed compaction. Architecture matters because LSM + merge + integration compose.

Advertisement

The architecture: every piece explained

The top strip is core. Table schema. LSM inside. Streaming ingest. Batch reads.

The middle row is features. CDC input. Merge engine. Change log stream. Snapshot + tag.

The lower rows are ops. Compaction service. Metrics. Ops — retention + concurrent writers.

Apache Paimon — LSM lakehouse + streaming ingest + updates + Flink integrationstreaming lakehouse with LSM insideTable schemaprimary key + fieldsLSM insidelevels + compactionStreaming ingestFlink connectorBatch readsSpark / Hive / TrinoCDC inputsupported nativelyMerge enginededup + partial updateChange log streamCDC outputSnapshot + tagtime travelCompaction servicebackgroundMetricscommit lag + filesOps — retention + concurrent writers + governancecdcmergeemittravelbackgroundwatchwatchoperateoperate
Apache Paimon LSM lakehouse for streaming ingest.
Advertisement

End-to-end flow

End-to-end: Flink stream ingests CDC to Paimon. LSM writes; merge engine deduplicates on PK. Compaction consolidates. Batch reads via Spark; streaming reads for downstream via change-log output. Snapshot for time travel.