Databases

Databases

Postgres internals, isolation levels, replication, pgvector, DuckDB, CDC.

68Articles
68Topics covered
Articles in this category

All 66 articles, sorted alphabetically

ARTICLE · 01

ClickHouse, in depth: MergeTree parts and sparse indexes, sort-key design, the engine family, materialized views, replication with Keeper and the too-many-parts failure

How ClickHouse stores and queries data: MergeTree parts, granules, marks and the sparse primary index, choosing ORDER BY and PARTITION BY, Replacing, …

Read article →
ARTICLE · 02

CockroachDB Architecture Deep Dive, in depth: ranges, Raft and leases, intents and parallel commits, clock uncertainty, retries and schema design

How CockroachDB works from SQL down to disk: key encoding, ranges and meta addressing, Raft per range and leader leases, write intents, transaction re…

Read article →
ARTICLE · 03

Database Connection Pooling

A deep dive on database connection pooling: why connections are expensive, connection lifecycle and overhead, pooling modes (statement, transaction, s…

Read article →
ARTICLE · 04

Database Indexing, in depth: choosing a strategy from access patterns, primary keys, tenants, pagination, time series and engine architecture

An indexing strategy for real workloads: how heap, clustered and LSM engines change index costs, choosing primary keys (sequential, UUIDv4, UUIDv7), t…

Read article →
ARTICLE · 05

ACID Transactions, in depth: what each letter promises, the mechanism behind it, the settings that quietly weaken it and how to test it

ACID transactions explained from first principles: atomicity through logs and undo, consistency as constraints plus your invariants, isolation levels …

Read article →
ARTICLE · 06

Bitmap index architecture

Deep-dive on bitmap indexes: one bitmap per distinct value, boolean AND/OR/NOT and popcount query evaluation, compression with RLE/Roaring, where they…

Read article →
ARTICLE · 07

Bloom filter architecture in LSM engines

Deep-dive on Bloom filters in log-structured merge-tree databases: how a compact bit array and k hash functions answer 'definitely absent' o…

Read article →
ARTICLE · 08

Buffer pool architecture

Deep-dive on database buffer pools: page tables and frames, pin counts and latches, clock sweep and midpoint LRU scan resistance, the WAL-before-data …

Read article →
ARTICLE · 09

CDC via logical decoding

Deep-dive on Change Data Capture via logical decoding: WAL-based change streaming, logical decoding of physical records, replication slots and output …

Read article →
ARTICLE · 10

Columnar Database Architecture in Depth: storage layout, encodings, pruning, vectorised execution and the write path

How a column-store database engine works inside: the IO arithmetic that motivates it, parts and granules, dictionary, run-length, delta and bit-packed…

Read article →
ARTICLE · 11

Connection pooling architecture

Deep-dive on database connection pooling: why connections are expensive, application pools vs proxy poolers, session vs transaction pooling trade-offs…

Read article →
ARTICLE · 12

Database deadlock detection architecture

Deep-dive on deadlock handling in databases: the lock manager and wait-for graph, eager vs lazy cycle detection, victim-selection policies, timeout fa…

Read article →
ARTICLE · 13

GIN inverted index architecture

Deep-dive on the Generalized Inverted Index (GIN): extracting many keys per composite value, mapping each key to a sorted posting list of row ids with…

Read article →
ARTICLE · 14

Group commit architecture

Deep-dive on database group commit: why one fsync per transaction caps throughput at flush latency, how the commit queue and group leader let many com…

Read article →
ARTICLE · 15

Hash join architecture

Deep-dive on the hash join: the algorithm that makes equality joins on large, unindexed relations fast by building a hash table on the smaller input a…

Read article →
ARTICLE · 16

Index-only scan architecture, in depth

How index-only scans really work: PostgreSQL's visibility map, heap fetches, INCLUDE columns, planner limits, the recent-data trap, InnoDB's…

Read article →
ARTICLE · 17

B-tree Indexes, in depth: page layout, fan-out arithmetic, search and split paths, concurrency and keeping indexes healthy

How the B+tree behind almost every database index works: why pages and fan-out keep trees three or four levels deep, what lives on leaf and internal p…

Read article →
ARTICLE · 18

Hash Indexes, in depth: bucket addressing, linear and extendible hashing, PostgreSQL internals, and when a hash index beats a B-tree

How hash indexes work from first principles: hashing keys to buckets, overflow chains, extendible and linear hashing with a runnable simulator, the on…

Read article →
ARTICLE · 19

Database indexing architecture, in depth: how indexes attach to storage, what each one costs on write, and how to design composite and covering indexes

An engine-agnostic guide to how database indexes are wired into storage: B+tree fanout and height, heap plus TID versus clustered primary keys, double…

Read article →
ARTICLE · 20

Database Isolation Levels, in depth: from the ANSI phenomena to dependency graphs, and how to test what your database really does

Isolation levels from their definitions: histories and serializability, the three ANSI SQL-92 phenomena and why they are ambiguous, the 1995 critique …

Read article →
ARTICLE · 21

LSM-tree storage engine architecture

Deep-dive on the Log-Structured Merge tree: WAL-plus-MemTable write path, immutable SSTables with bloom filters and block index, L0/L1..Ln level struc…

Read article →
ARTICLE · 22

Materialized views -- precomputed query results

Deep-dive on database materialized views: the expensive-query need, precomputing and storing results, materialized vs regular view (stored vs recomput…

Read article →
ARTICLE · 23

MVCC -- readers and writers that never block each other

Deep-dive on Multi-Version Concurrency Control (MVCC): the reader-writer contention problem, keeping multiple versions, consistent snapshots, no read …

Read article →
ARTICLE · 24

OLTP vs OLAP Database Architectures in Depth

OLTP vs OLAP in depth: row stores, B+trees, WAL and MVCC versus columnar encoding, pruning and vectorised execution, a worked orders example, CDC pipe…

Read article →
ARTICLE · 25

Database Partitioning Architecture, in depth: range, list, hash and composite schemes, partition pruning, partitioned indexes and the retention lifecycle

A first-principles guide to single-node declarative table partitioning in PostgreSQL and MySQL: when partitioning helps and when it hurts, choosing th…

Read article →
ARTICLE · 26

Database query planner architecture, in depth: statistics, cost models, join search and why plans go wrong

How a relational query planner turns SQL into an execution plan, using PostgreSQL as the running example: parse and rewrite, access paths, the cost mo…

Read article →
ARTICLE · 27

Database Replication Architecture in Depth: What Gets Shipped, When a Commit Is Acknowledged, and How Failover Loses Data

Database replication from first principles: statement, row and physical log shipping; asynchronous, semi-synchronous, synchronous and quorum commit wi…

Read article →
ARTICLE · 28

Serializable isolation, in depth: how databases enforce it, what it costs, and how to run it in production

How serializable isolation is enforced and what it costs: two-phase locking, serializable snapshot isolation and optimistic validation compared by who…

Read article →
ARTICLE · 29

Database sharding -- horizontal partitioning across nodes

Deep-dive on database sharding: the single-node limit, the critical shard key choice, sharding strategies (hash/range/directory), routing, hot shards …

Read article →
ARTICLE · 30

Snapshot isolation architecture, in depth: how snapshots are built, the commit check, write skew, SSI and distributed timestamps

How snapshot isolation works inside a database: building a snapshot from transaction ids, the visibility rule, first-committer-wins and first-updater-…

Read article →
ARTICLE · 31

Transaction isolation architecture, in depth: what each engine's levels really mean, reproducible anomalies and retry-safe code

A practical guide to transaction isolation: the anomalies that define each level, how locking, snapshot isolation and serializable snapshot isolation …

Read article →
ARTICLE · 32

Transactional Database Architecture in Depth

A 2500-word walkthrough of transactional DB: MVCC, lock manager, WAL/redo, rollback, isolation levels, 2PC, vacuum, recovery.

Read article →
ARTICLE · 33

Two-phase commit architecture

Deep-dive on two-phase commit: the prepare and decide phases, write-ahead logging on coordinator and participants, held locks and the in-doubt window,…

Read article →
ARTICLE · 34

Vacuum and bloat architecture

Deep-dive on Postgres vacuum and bloat: why MVCC updates leave dead tuples, the cluster-wide visibility horizon and the idle transactions, replication…

Read article →
ARTICLE · 35

Vector database architecture, in depth: index families, memory math, quantization with rescoring and measured recall

How a vector database works inside: write path and segments, flat, IVF, HNSW and SSD graph indexes, per-vector memory math, int8, PQ and binary quanti…

Read article →
ARTICLE · 36

WAL architecture, in depth: the two logging rules, LSNs, checkpoints, ARIES-style recovery worked by hand, torn pages and fsync failure

How a write-ahead log makes a database durable and fast: the log-before-data and log-before-commit rules, record format and LSNs, steal and no-force, …

Read article →
ARTICLE · 37

Write amplification architecture

Deep-dive on write amplification in LSM storage engines: the WAL/memtable/flush path, leveled vs tiered compaction rewrites, the read/write/space trad…

Read article →
ARTICLE · 38

DuckDB, in depth: the in-process engine, vectorised push-based pipelines, single-file storage, MVCC and out-of-core execution

How DuckDB works inside and what that means in operation: the in-process model, the query pipeline, 2,048-value vectors, morsel-driven parallel pipeli…

Read article →
ARTICLE · 39

DuckDB Practical Uses, in depth: Parquet and CSV analysis, dataframes, lake queries, ETL, CI data tests, and where it stops fitting

Practical DuckDB from first principles: the in-process vectorized design, querying Parquet and CSV in place, zero-copy interop with pandas, Polars and…

Read article →
ARTICLE · 40

Postgres Logical Replication Use Cases, in depth: upgrades, filtered feeds, consolidation, tenant moves, bidirectional sync and their limits

A use-case catalogue for PostgreSQL logical replication: how publications, slots and apply workers fit together, then a recipe per case (near-zero-dow…

Read article →
ARTICLE · 41

Materialized Views in Postgres, in depth: plain vs concurrent refresh, locking, scheduling and incremental alternatives

How PostgreSQL materialized views really work: storage and populated state, what plain REFRESH and REFRESH CONCURRENTLY do internally and which locks …

Read article →
ARTICLE · 42

MongoDB, in depth: inside the engine, from the WiredTiger cache and checkpoints to the query planner, the oplog, elections and sharded routing

How MongoDB works underneath the document API: the WiredTiger cache, MVCC snapshots, journal and checkpoints, how the query planner picks and caches p…

Read article →
ARTICLE · 43

MongoDB: When It Actually Fits, in depth: the document as the unit of atomicity, the aggregate test, modelling, consistency settings and where it strains

A first-principles fit test for MongoDB: what the document model, replica sets and sharding actually guarantee, the aggregate-shaped access test, work…

Read article →
ARTICLE · 44

MySQL InnoDB, in depth: one UPDATE through the clustered index, buffer pool, undo, redo, doublewrite and binlog, plus read views, next-key locking and crash recovery

How InnoDB executes and protects a single row change: the clustered index and hidden columns, the buffer pool, undo records and MVCC read views with t…

Read article →
ARTICLE · 45

MySQL Replication, in depth: binlog formats, GTIDs, async versus semi-sync, parallel appliers, lag you can trust and safe failover

How MySQL replication works and how to run it: the binary log, row versus statement formats, file positions versus GTIDs, asynchronous and semi-synchr…

Read article →
ARTICLE · 46

MySQL vs Postgres in 2026, in depth: storage, MVCC, locking, replication and the release lines that matter now

An engine-level comparison of MySQL and PostgreSQL in 2026: the current release lines, clustered InnoDB versus the Postgres heap, undo-log purge versu…

Read article →
ARTICLE · 47

pgvector at 100M Scale, in depth: sizing, halfvec and binary quantization, index builds, filtered search, partitioning and replicas

How to run pgvector with a hundred million embeddings: storage and memory arithmetic, index dimension limits, halfvec and binary-quantized indexes wit…

Read article →
ARTICLE · 48

PostgreSQL WAL, in depth: segments and LSNs, synchronous_commit, full-page writes, checkpoint sizing, archiving and a full pg_wal

How PostgreSQL's write-ahead log works in practice: how LSNs map to 16 MB segment files, what pg_waldump shows, the wal_level and synchronous_com…

Read article →
ARTICLE · 49

Postgres Index Strategies, in depth: designing indexes from the workload, proving them with EXPLAIN and retiring them safely

A workload-first method for PostgreSQL indexing on one worked table: finding hot query shapes with pg_stat_statements, composite column order, partial…

Read article →
ARTICLE · 50

PostgreSQL Indexes

The four PostgreSQL index types explained: B-tree for range queries and sorting, GIN for inverted indexes and full-text search, GIST for spatial and c…

Read article →
ARTICLE · 51

Postgres Isolation Levels, in depth: per-statement snapshots, the UPDATE re-check, 40001 errors, SERIALIZABLE in practice and a coupon-cap worked example

What each PostgreSQL isolation level actually does: Read Committed's per-statement snapshots and wait-and-recheck updates, Repeatable Read's…

Read article →
ARTICLE · 52

PostgreSQL Logical Replication + CDC

PostgreSQL logical replication: row-level selective replication across major versions, WAL-based change data capture, replication slots and decoding, …

Read article →
ARTICLE · 53

PostgreSQL MVCC and VACUUM, in depth: tuple versions, the xmin horizon, autovacuum arithmetic and freezing

How PostgreSQL implements MVCC in the heap and how VACUUM cleans up after it: tuple headers and pageinspect, snapshot visibility, the xmin horizon and…

Read article →
ARTICLE · 54

PostgreSQL Table Partitioning, in depth: declarative DDL, the lock each command takes, building indexes without downtime, migrating a live table and keeping the planner honest

An operational guide to PostgreSQL declarative partitioning: how a partitioned table routes rows, the DDL and bound rules, what ATTACH, DETACH and ind…

Read article →
ARTICLE · 55

Redis Data Structures, in depth: choosing types, internal encodings, queues and streams, big keys and cluster limits

A practical guide to Redis data structures: strings, hashes, lists, sets, sorted sets, streams, bitmaps and HyperLogLog, the compact and full encoding…

Read article →
ARTICLE · 56

Redis, in depth: the single command thread, keyspace and expiry, eviction, atomicity, replication, Sentinel, Cluster and latency diagnosis

How a Redis server actually runs: the event loop and io-threads, keyspace hashing and incremental rehash, lazy and active expiry, maxmemory and sample…

Read article →
ARTICLE · 57

Redis Persistence Modes, in depth: RDB snapshots, AOF and fsync policies, multi-part AOF, fork costs, WAITAOF and choosing what you can afford to lose

How Redis persists an in-memory dataset: RDB snapshots and fork with copy-on-write, the append-only file and its three fsync policies, rewrites and th…

Read article →
ARTICLE · 58

ScyllaDB vs Cassandra, in depth: shard-per-core against the JVM, tablets against vnodes, compaction, licensing and how to choose

An engineer's comparison of ScyllaDB and Apache Cassandra: the shared CQL and LSM foundation, the threading and memory models that make them beha…

Read article →
ARTICLE · 59

Sharding Strategies Compared, in depth: range, hash, token ring, fixed slots, directory and entity sharding, and what each costs to reshard

A side-by-side comparison of database sharding strategies: the two-level key-to-shard and shard-to-node mapping, range sharding with automatic splits,…

Read article →
ARTICLE · 60

SQL Query Optimization, in depth: finding the queries that matter, reading EXPLAIN ANALYZE, and a catalogue of fixes that actually work

A practical PostgreSQL-centred method for SQL query optimization: rank queries with pg_stat_statements, read EXPLAIN ANALYZE and BUFFERS output, fix n…

Read article →
ARTICLE · 61

TimescaleDB, in depth: hypertables and chunks, chunk exclusion, sizing the chunk interval, the columnstore, continuous aggregate refresh and the job scheduler

How TimescaleDB works inside Postgres: hypertables and chunks, plan-time and run-time chunk exclusion, sizing the chunk interval with a worked example…

Read article →
ARTICLE · 62

TimescaleDB vs InfluxDB, in depth: storage engines, schema, write and query paths, cardinality, compression, rollups and how to choose

A version-explicit comparison of TimescaleDB and InfluxDB for time-series workloads: hypertables and chunks versus TSM and the InfluxDB 3 Arrow, DataF…

Read article →
ARTICLE · 63

TimescaleDB vs InfluxDB vs Prometheus, in depth: data models, storage engines, query languages and a sizing worked example

A first-principles comparison of TimescaleDB, InfluxDB 3 and Prometheus: how each models series, how its storage engine moves data from WAL to compres…

Read article →
ARTICLE · 64

When to Pick Cassandra, in depth: the architecture behind the choice, where it fits, where it hurts and how to decide

A decision guide to Apache Cassandra: how its leaderless ring and LSM write path shape what is cheap and expensive, query-first modelling, the workloa…

Read article →
ARTICLE · 65

When to Pick DynamoDB, in depth: access patterns first, where it fits, where it hurts and how to decide

A decision guide for Amazon DynamoDB: what you trade for serverless scale, an access-pattern inventory method, a worked multi-tenant order system with…

Read article →
ARTICLE · 66

When to Pick Postgres, in depth: the architecture behind the default choice, where it fits, where it strains and how to decide

A decision guide for PostgreSQL grounded in its architecture: one write primary, MVCC with vacuum, process-per-connection and an extensible type and i…

Read article →