Why architecture matters here
The architecture matters because the naive alternative — a permanently-running Spark cluster plus ad-hoc scripts — fails on cost, coherence, and maintenance at once. A long-lived cluster bills for every idle hour, and analytics workloads are spiky, so utilization is poor and the bill is high. Scripts that each hardcode their own schema and output layout drift apart, so two pipelines reading 'the same' data disagree about its columns. And someone has to patch, scale, and restart the cluster. Glue removes all three: no idle cost, one catalog everyone shares, and no servers to tend.
The serverless model is the headline economic change. Because DPUs are provisioned per job and released at the end, you pay for computation, not for capacity kept warm 'just in case'. For workloads that run a few times a day — the shape of most ETL — that is dramatically cheaper than a cluster sized for peak and idle the rest of the time. It also removes an entire class of operational toil: no cluster to right-size continuously, no node failures to recover, no version upgrades to schedule. The tradeoff is cold-start latency and less control, which is why Glue fits batch ETL far better than low-latency interactive compute.
The catalog-as-contract is the coherence change, and it is what makes the lake usable by more than one team. When Athena, Redshift Spectrum, and EMR all resolve a table through the same Glue catalog, a schema change made once is seen everywhere, and governance — column-level permissions via Lake Formation, for instance — is enforced at one place rather than reinvented per engine. The catalog turns a lake from a swamp of files that each consumer interprets privately into a set of governed tables with a single source of truth for structure and access.
Incrementality is the third pillar, and it is where the bookmark earns its keep. Reprocessing an entire multi-terabyte dataset on every run is both slow and expensive, and it scales the wrong way — the more history you accumulate, the longer every run takes. Job bookmarks let a job persist a marker of what it has already consumed (by partition, by primary key, by timestamp) so the next run touches only new data. That converts a job whose cost grows with total history into one whose cost grows with the delta since last run, which is the only model that stays affordable as a dataset ages. Together — serverless economics, a shared catalog, and bookmarked incrementality — these are the reasons Glue is an architecture worth understanding rather than just a managed Spark button, and each addresses a distinct way that hand-rolled lake ETL becomes unaffordable or incoherent over time.
The architecture: every piece explained
The crawler is the discovery mechanism. Pointed at an S3 prefix or a JDBC source, it samples the data, infers column names and types, detects the partition structure (a Hive-style year=/month=/day= layout, say), and registers or updates a table in the catalog. It can run on a schedule to pick up new partitions as data lands. The crawler is convenient but opinionated — it guesses types and can mis-infer — so many teams treat its output as a starting point and manage critical schemas explicitly.
The Data Catalog is the metadata store and the heart of the system. Organized as databases containing tables, each table records a schema, a serialization format, a physical location, and a list of partitions. It is Hive-metastore compatible, which is why every AWS analytics engine can read it. Crucially it stores metadata only — the data itself stays in S3 — so the catalog is cheap, and 'the table' is a pointer-plus-schema, not a copy of the bytes.
The Glue job is serverless Spark. You supply a script; Glue allocates DPUs (each a bundle of vCPU and memory), runs the Spark application across them, and releases them at completion. Worker types (standard, G.1X, G.2X) and the DPU count are the two knobs that set the job's parallelism and cost. Because provisioning is per-run, sizing is a per-job decision rather than a standing cluster commitment.
The DynamicFrame is Glue's schema-flexible data structure, a layer over the Spark DataFrame designed for the reality that lake data has inconsistent, evolving schemas. Where a DataFrame demands a fixed schema, a DynamicFrame tolerates records that disagree — a field that is sometimes a string and sometimes an int becomes a 'choice' type it can resolve later — and offers ETL-oriented transforms (ResolveChoice, ApplyMapping, Relationalize) for cleaning messy input. You convert to a DataFrame when you need full Spark SQL, and back when you need Glue's tolerance. The job bookmark persists per-job state — the high-water mark of what has been processed — so a rerun skips already-consumed data and handles only the delta; it is what makes a scheduled job incremental rather than a full reprocess. Triggers and workflows orchestrate jobs into pipelines: a trigger fires a job on a schedule, on an event, or on the completion of another job, and a workflow strings jobs and crawlers into a DAG with shared run state. Together these pieces — crawler, catalog, serverless job, DynamicFrame, bookmark, and workflow — are the full machine, each solving a specific piece of the raw-to-refined problem.
End-to-end flow
Trace a daily ingestion pipeline. Raw event files land in an S3 bucket under a date-partitioned prefix throughout the day. Overnight a scheduled crawler runs, samples the new partition, and updates the raw_events table in the catalog with the new day= partition and any schema changes it detects. The catalog now knows the day's data exists and where it lives, without any bytes having moved.
A trigger then fires the transform job. The Glue job starts, Glue provisions its DPUs, and the script reads raw_events through the catalog as a DynamicFrame. Because the job has a bookmark, it does not read the whole table — it resumes from its high-water mark and reads only the new partition. It applies mappings to normalize field names, resolves a 'choice' column whose type varies, filters malformed records to a rejects path, and joins against a reference table also defined in the catalog.
The job writes its output as partitioned Parquet to a curated S3 location and registers the new partitions in the curated_events catalog table. It then advances its bookmark so tomorrow's run starts after today's data. Glue tears down the DPUs; billing stops. The whole run touched only the day's delta, cost only the minutes it computed, and left the catalog updated so downstream consumers see the new data immediately.
Now the consumption side, which is where the shared catalog pays off. An analyst opens Athena and queries curated_events for the new day; Athena resolves the table through the same Glue catalog, prunes to the relevant partition, and scans only that Parquet — fast and cheap because the layout and schema are exactly what the job wrote. A Redshift Spectrum query and an EMR Spark job against the same table see identical structure. No engine carries its own copy of the schema; each is a client of the one catalog definition. If the transform job later adds a column, it updates the catalog once and every engine sees the new column on its next query. That single-source-of-truth behavior — the crawler and jobs writing the catalog, every query engine reading it — is what makes the end-to-end flow coherent rather than a set of pipelines that quietly drift out of agreement about what the data even is.