Hive + Impala (SQL on Hadoop)

Hive + Impala (SQL on Hadoop)

Deep technical articles on this topic.

150Articles
150Topics covered
Articles in this category

All 150 articles, sorted alphabetically

Advertisement
ARTICLE · 01

Cloudera Data Warehouse (CDW)

Impala as managed service on Cloudera.

Read article
ARTICLE · 02

Hive 3 ACID Improvements

Default managed = ACID.

Read article
ARTICLE · 03

Hive 4 Key Features

Iceberg + storage handler + SQL compliance.

Read article
ARTICLE · 04

Hive ACID Transactions

How Hive supports ACID transactions with delta files and lock manager, enabling INSERT, UPDATE, DELETE, and MERGE.

Read article
ARTICLE · 05

Hive ACID architecture

Deep-dive on Hive ACID transactions: write IDs and snapshot isolation, delta and delete-delta layout with hidden row IDs, MERGE mechanics, lock manage…

Read article
ARTICLE · 06

Hive Architecture Deep Dive

The internal architecture of Hive: how HiveServer2, the driver, compiler stages, and metastore cooperate to run SQL over HDFS.

Read article
ARTICLE · 07

Hive Authorization

How Hive authorization works: legacy storage-based vs SQL standard vs Ranger integration.

Read article
ARTICLE · 08

Hive Backup + Restore Strategies

How to back up Hive tables + metastore + restore.

Read article
ARTICLE · 09

Beeline vs Hive CLI

Modern (JDBC) vs legacy client.

Read article
ARTICLE · 10

Hive Bucketing

How Hive bucketing hashes rows into a fixed number of files, and when it enables sort-merge bucket joins for big-to-big joins.

Read article
ARTICLE · 11

Hive bucketing

Deep-dive on Hive bucketing: CLUSTERED BY hashing into fixed bucket files, bucket map and sort-merge-bucket joins without shuffle, TABLESAMPLE samplin…

Read article
ARTICLE · 12

Hive Bucketing for Efficient Joins

SMB — sort-merge-bucket.

Read article
ARTICLE · 13

Hive Table Buckets

Hash-based sub-partition. Efficient joins + sampling.

Read article
ARTICLE · 14

Hive Cost-Based Optimizer (Calcite)

Choose plan based on stats.

Read article
ARTICLE · 15

Hive CBO architecture

Deep-dive on Hive's cost-based optimizer: Calcite RelNode pipeline, predicate pushdown and partition pruning, NDV-based cardi…

Read article
ARTICLE · 16

Columnar Advantages for Hive

Projection + compression + vectorized reads.

Read article
ARTICLE · 17

Hive ACID Compaction

How the ACID compactor merges delta files into base files, why it's a separate service from HDFS compaction, and how to tune it.

Read article
ARTICLE · 18

Hive Complex Types

Nested data support.

Read article
ARTICLE · 19

Hive Compression Options

Snappy, GZ, Zstd, LZ4 codecs.

Read article
ARTICLE · 20

Hive Deprecated Features

Indexes, MR engine, some SerDes.

Read article
ARTICLE · 21

Hive dynamic partitioning

Deep-dive on Hive dynamic partitioning: static vs dynamic keys and trailing-column binding, DISTRIBUTE BY for file-count control, max-partition guardr…

Read article
ARTICLE · 22

Hive Execution Engines

MR, Tez, Spark. Choose per workload.

Read article
ARTICLE · 23

Hive File Formats

The file formats Hive supports and their trade-offs: row-based vs columnar, compression, schema evolution, splittability.

Read article
ARTICLE · 24

Hive Future

Hive's role shifting.

Read article
ARTICLE · 25

HiveServer2 architecture

Deep-dive on HiveServer2: Thrift transports and session/operation managers, async execution over Tez session pools and LLAP, result fetch and spooling…

Read article
ARTICLE · 26

Apache Iceberg -- the open table format for the lakehouse

Deep-dive on Apache Iceberg: layered metadata (catalog/manifest/data files), immutable snapshots, ACID via atomic pointer swaps, time travel, hidden p…

Read article
ARTICLE · 27

Hive → Iceberg Migration

Convert existing tables. Metadata rewrite.

Read article
ARTICLE · 28

Hive + Iceberg Tables

Modern table format on Hive.

Read article
ARTICLE · 29

Hive vs Impala

Direct comparison of Hive vs Impala for different workloads.

Read article
ARTICLE · 30

Hive Indexes (Deprecated)

Legacy. Use partitioning + bucketing.

Read article
ARTICLE · 31

Hive Join Strategy Selection

Auto-select + hints for control.

Read article
ARTICLE · 32

Hive Joins

Join strategies + hints.

Read article
ARTICLE · 33

Hive with JSON + Avro Data

SerDes + complex types.

Read article
ARTICLE · 34

Hive JVM Tuning

HS2 + metastore + Tez AM.

Read article
ARTICLE · 35

Hive LLAP

How LLAP (Live Long And Process) makes Hive interactive by running long-lived daemons that cache data and skip container startup entirely.

Read article
ARTICLE · 36

Hive LLAP architecture

Deep-dive on Hive LLAP: HiveServer2 planning, Tez AM coordination, daemon executor slots, async IO elevator, off-heap columnar cache with SSD tier, Zo…

Read article
ARTICLE · 37

Hive LLAP Deep Dive

How Hive LLAP daemons work: shared in-memory cache, query execution, integration with Tez.

Read article
ARTICLE · 38

Hive LLAP Performance Deep

Cached daemons + fragment scheduling.

Read article
ARTICLE · 39

Hive Materialized Views

Precomputed aggregates + auto-rewrite.

Read article
ARTICLE · 40

Hive Metastore

How the Hive Metastore stores tables, partitions, and statistics, and why it has become the de facto schema registry for Spark, Presto, Impala, and Tr…

Read article
ARTICLE · 41

Hive Metastore architecture

Deep-dive on the Hive Metastore: Thrift API over MySQL/Postgres, partition pruning paths, column statistics for CBOs, ACID transaction state, event no…

Read article
ARTICLE · 42

Hive Metastore Backup

Backing DB regular backups.

Read article
ARTICLE · 43

Hive Metastore HA

How to run Hive Metastore in HA configuration.

Read article
ARTICLE · 44

Hive Monitoring

HS2 metrics + query logs.

Read article
ARTICLE · 45

Hive on Spark

Spark as engine. Alternative to Tez.

Read article
ARTICLE · 46

Hive on Tez Deep Dive

DAG execution + optimizations.

Read article
ARTICLE · 47

Hive Cost-Based Optimizer

How Hive's cost-based optimizer chooses join orders, join strategies, and plan alternatives based on table and column statistics.

Read article
ARTICLE · 48

ORC format architecture

Deep-dive on ORC internals: file/stripe/stream layout, RLEv2 and dictionary encodings, row-group indexes with seek positions, min/max stats and bloom …

Read article
ARTICLE · 49

Hive ORC Format Deep Dive

How ORC (Optimized Row Columnar) format works internally.

Read article
ARTICLE · 50

ORC Format

The internal structure of ORC files: stripes, row groups, indexes, and how column pruning + predicate pushdown work together.

Read article
ARTICLE · 51

Apache Hive Overview

What Hive is, why SQL over HDFS mattered, and where Hive fits in modern data platforms alongside Spark, Presto, and Impala.

Read article
ARTICLE · 52

Hive Parquet Format Deep Dive

How Parquet compares to ORC and its internal structure.

Read article
ARTICLE · 53

Parquet Format

The internal structure of Parquet: row groups, column chunks, page indexes, dictionary encoding, and cross-engine compatibility.

Read article
ARTICLE · 54

Hive Partition Evolution

Add + drop partitions. Repair.

Read article
ARTICLE · 55

Hive Partition Pruning

Query planner drops non-matching partitions.

Read article
ARTICLE · 56

Hive Partitioning

How Hive partitioning works via HDFS directory layout, why it drives query performance, and how to design partitions right.

Read article
ARTICLE · 57

Hive Table Partitions

Directory-based partitioning. Prune scans.

Read article
ARTICLE · 58

Hive Predicate Pushdown

Filter at storage layer. ORC/Parquet stats.

Read article
ARTICLE · 59

Hive Query Execution

How Hive queries execute end-to-end: parse, plan, optimize, generate DAG, execute on Tez/Spark, and stream results.

Read article
ARTICLE · 60

Hive replication architecture

Deep-dive on Hive replication: event-driven incremental replication over the notification log, bootstrap and checkpointed cycles, DistCp data movement…

Read article
ARTICLE · 61

Hive Security

How Hive integrates with Kerberos + LDAP for auth, and encryption at rest + in transit.

Read article
ARTICLE · 62

Hive Security

Auth + authz.

Read article
ARTICLE · 63

Hive SerDes

Row format for different sources.

Read article
ARTICLE · 64

Hive small-file problem

Deep-dive on the small-file problem: sources (streaming, dynamic partitioning, appends), the three-layer cost (metadata, task overhead, read amplifica…

Read article
ARTICLE · 65

Hive on Spark Engine

How Hive can run on Spark instead of Tez.

Read article
ARTICLE · 66

Hive Storage Handlers

Query non-HDFS data — HBase, JDBC, ES.

Read article
ARTICLE · 67

Hive Streaming Ingestion

Real-time inserts to Hive.

Read article
ARTICLE · 68

Hive Subqueries + CTEs

Common patterns + limits.

Read article
ARTICLE · 69

Hive Table Statistics

ANALYZE TABLE. CBO uses stats.

Read article
ARTICLE · 70

Hive Tables

CREATE TABLE. Storage lifecycle differs.

Read article
ARTICLE · 71

Hive Table Types Summary

Managed, external, ACID, temporary, materialized view.

Read article
ARTICLE · 72

Hive on Tez

Why Tez replaced MapReduce as Hive's execution engine, how it uses DAGs instead of MR sequences, and how container reuse cuts que…

Read article
ARTICLE · 73

Hive on Tez architecture

Deep-dive on Tez execution for Hive: vertices and typed edges vs MapReduce chains, the per-session application master, YARN container reuse, runtime a…

Read article
ARTICLE · 74

Hive Tez Optimization

DAG execution. Container reuse.

Read article
ARTICLE · 75

Tez UI + Application Timeline Server

Query DAG visualization.

Read article
ARTICLE · 76

Hive Transactional (ACID) Tables

Row-level UPDATE/DELETE. Delta files.

Read article
ARTICLE · 77

Hive ACID Transactions Deep Dive

How Hive ACID transactions work: managed tables, delta files, compaction, isolation.

Read article
ARTICLE · 78

Hive TRANSFORM with Python

Streaming rows through script.

Read article
ARTICLE · 79

Hive Query Tuning Playbook

EXPLAIN + partition + join + engine.

Read article
ARTICLE · 80

Hive UDAF (User-Defined Aggregate)

Custom aggregation logic.

Read article
ARTICLE · 81

Hive UDFs and UDAFs

Deep-dive on Hive user-defined functions: UDF/GenericUDF/UDAF/UDTF types, ObjectInspectors, vectorized implementations vs row-mode fallback, UDAF part…

Read article
ARTICLE · 82

Hive UDFs

How to write Hive UDFs, UDAFs, and UDTFs, when to use them, and the performance and safety implications.

Read article
ARTICLE · 83

Hive UDTF (User-Defined Table Function)

Row → many rows. EXPLODE built-in.

Read article
ARTICLE · 84

Hive Upgrade Path

Version compat + metastore schema migration.

Read article
ARTICLE · 85

Hive Vectorization

Process batches of rows. Massive perf.

Read article
ARTICLE · 86

Hive vectorized execution

Deep-dive on Hive's vectorized engine: VectorizedRowBatch and ColumnVector anatomy, expression templates and monomorphic loop…

Read article
ARTICLE · 87

Hive View Types

The main Hive view types: virtual (logical), materialized (physical), and their trade-offs.

Read article
ARTICLE · 88

Hive Views and Materialized Views

How Hive views work (query rewriting on-the-fly), how materialized views cache results, and when each is the right choice.

Read article
ARTICLE · 89

Hive vs Presto/Trino

Batch vs interactive.

Read article
ARTICLE · 90

Hive Window Functions

ROW_NUMBER, RANK, LAG, LEAD.

Read article
ARTICLE · 91

Hive window functions

Deep-dive on Hive window functions: the OVER clause with PARTITION BY, ORDER BY, and frames, ranking/aggregate/positional function families, ROWS-vs-R…

Read article
ARTICLE · 92

HiveQL Syntax Essentials

SQL-like with big-data extensions.

Read article
ARTICLE · 93

HiveServer2 (HS2)

Thrift server. Multi-session.

Read article
ARTICLE · 94

HiveServer2 Clustering

Multiple HS2 + ZK URI.

Read article
ARTICLE · 95

Impala 4 Key Features

Iceberg + Kudu + admission improvements.

Read article
ARTICLE · 96

Impala Administration

Impala admin tasks: rolling restart, log analysis, tuning.

Read article
ARTICLE · 97

Impala Admission Control

How Impala admission control queues queries when resources are constrained, and how to configure per-pool limits.

Read article
ARTICLE · 98

Impala admission control architecture

Deep-dive on Impala admission control: resource pools, per-host memory estimates vs MEM_LIMIT, statestore-gossiped local decisions, the admissiond cen…

Read article
ARTICLE · 99

Impala Analytic Functions

How Impala's window functions (RANK, LAG, LEAD, SUM OVER) enable analytic queries.

Read article
ARTICLE · 100

Impala Architecture

The three daemon types in Impala, how they cooperate, and what each is responsible for.

Read article
ARTICLE · 101

Impala Cost-Based Optimizer

Uses table stats. Choose plan.

Read article
ARTICLE · 102

Impala Cluster Sizing

Memory + concurrent queries + workload.

Read article
ARTICLE · 103

Impala Complex Types

Nested Parquet reads.

Read article
ARTICLE · 104

Impala COMPUTE STATS

Table + column + incremental stats.

Read article
ARTICLE · 105

Impala Configuration + Admin

Common tuning knobs.

Read article
ARTICLE · 106

Impala Coordinator + Executor Roles

Split roles for scale.

Read article
ARTICLE · 107

Impala Cost Management

Right-sizing + auto-scale + admission.

Read article
ARTICLE · 108

Impala Cost Optimization

How to reduce Impala compute + storage costs.

Read article
ARTICLE · 109

Impala Daemons Deep Dive

How the Statestore's pub/sub topics work, how the Catalog syncs with the Hive Metastore, and how metadata invalidation propagates…

Read article
ARTICLE · 110

Impala DML Support

INSERT, UPDATE, DELETE (Kudu).

Read article
ARTICLE · 111

Impala Execution Model

How Impala executes queries: vectorized operators, LLVM code generation, streaming data flow between fragments.

Read article
ARTICLE · 112

Impala Future

Where Impala fits in modern stack.

Read article
ARTICLE · 113

Impala on HDFS + S3

Native readers for both.

Read article
ARTICLE · 114

Migrate from Hive to Impala

Query rewrite + perf gains.

Read article
ARTICLE · 115

Impala + Iceberg Tables

Modern table format support.

Read article
ARTICLE · 116

Impala + Iceberg Migration Path

Modernize table format.

Read article
ARTICLE · 117

Impala Daemons

Distributed roles.

Read article
ARTICLE · 118

Impala In-Memory Execution

Everything in RAM. Spill on OOM.

Read article
ARTICLE · 119

Impala INVALIDATE vs REFRESH

Metadata cache management.

Read article
ARTICLE · 120

Impala Join Strategies

Broadcast vs partitioned.

Read article
ARTICLE · 121

Impala + Kudu Deep Dive

How Impala integrates with Kudu for real-time analytical + operational.

Read article
ARTICLE · 122

Impala + Kudu

How Impala works with Apache Kudu to give both fast scans and low-latency updates, and when to choose Kudu over HDFS.

Read article
ARTICLE · 123

Impala LLVM Code Generation

Query-specific compiled operators.

Read article
ARTICLE · 124

Impala Memory Limits

Per-query + per-node caps.

Read article
ARTICLE · 125

Impala Metadata Deep Dive

How Impala's catalog service + Impalad local metadata cache work together.

Read article
ARTICLE · 126

Impala Monitoring

Query profile + web UI + Prometheus.

Read article
ARTICLE · 127

Impala MPP Architecture

Shared-nothing distributed execution.

Read article
ARTICLE · 128

Impala Node Types

Coordinator-only + executor-only.

Read article
ARTICLE · 129

Impala ORC Support

Growing but historically Parquet-first.

Read article
ARTICLE · 130

Apache Impala Overview

What Impala is, how MPP execution makes it interactive, and where it fits versus Hive, Presto, and Spark SQL.

Read article
ARTICLE · 131

Impala Parquet Native Reader

Optimized columnar. Predicate pushdown.

Read article
ARTICLE · 132

Impala Partition Pruning

Skip non-matching partitions.

Read article
ARTICLE · 133

Impala query execution architecture

Deep-dive on Impala's MPP execution: coordinator planning, statestore and catalog daemons, admission control pools, pipelined…

Read article
ARTICLE · 134

Impala Query Hints

Override planner decisions.

Read article
ARTICLE · 135

Impala Query Planning

How Impala plans queries: parse, analyze, single-node plan, distributed plan, fragment scheduling.

Read article
ARTICLE · 136

Impala Query Plans

How to read Impala's EXPLAIN output to understand and optimize queries.

Read article
ARTICLE · 137

Impala Query Queue

Admission control per resource pool.

Read article
ARTICLE · 138

Impala + Ranger + Kerberos

Enterprise security integration.

Read article
ARTICLE · 139

Impala + Ranger Integration

How Impala integrates with Ranger for centralized authorization.

Read article
ARTICLE · 140

Impala Runtime Filters

Dynamic filter propagation.

Read article
ARTICLE · 141

Impala Scale

How Impala scales to hundreds of nodes: coordinators, statestore, workload management.

Read article
ARTICLE · 142

Impala Shell + Web UI

Interactive + monitoring.

Read article
ARTICLE · 143

Impala Spill to Disk

OOM protection via disk spill.

Read article
ARTICLE · 144

Impala Table Statistics

How Impala uses table and column statistics for the cost-based optimizer, and how to keep stats fresh.

Read article
ARTICLE · 145

Impala Text + CSV Support

Basic external data.

Read article
ARTICLE · 146

Impala Troubleshooting

The common Impala issues (slow queries, OOM, metadata staleness) and how to fix.

Read article
ARTICLE · 147

Impala Upgrade Path

Rolling + version compat.

Read article
ARTICLE · 148

Impala vs Hive

The trade-offs between Impala and Hive: latency, concurrency, workload compatibility, ecosystem fit.

Read article
ARTICLE · 149

Impala vs Presto/Trino

Feature + perf comparison.

Read article
ARTICLE · 150

Impala vs Snowflake

Self-managed vs cloud SaaS.

Read article