Hive Complex Types, in depth: ARRAY, MAP, STRUCT, LATERAL VIEW and modelling nested data

The four types and what each one models

A complex type is a column whose value is itself structured. Hive has four, and each answers a different modelling question.

The types nest freely: ARRAY<STRUCT<sku:STRING,qty:INT>> is a list of records, and MAP<STRING,ARRAY<INT>> maps a key to a list. The one hard restriction is that a map key must be a primitive type; the value can be anything.

A STRUCT has a schema: every row has the same field names, each with its own type, so addr.zip is checked at compile time. A MAP has no schema for its keys: any row can carry any key, all values share one type, and attrs['colour'] with a typo silently returns NULL. Use a STRUCT when you know the fields and a MAP when the key set is open-ended.

UNIONTYPE exists, but support across formats, functions and engines has always been partial, and Impala does not support it. Read it from legacy data; do not design new tables around it. A STRUCT with one nullable field per alternative is the portable equivalent.

Declaring and constructing complex values

Here is the order table used through the rest of the article. It is stored as ORC; the storage section explains why that matters more for nested data than for flat data.

CREATE TABLE sales.orders (
  order_id   BIGINT,
  order_ts   TIMESTAMP,
  customer   STRUCT<id:BIGINT, tier:STRING, region:STRING>,
  items      ARRAY<STRUCT<sku:STRING, qty:INT, unit_price:DECIMAL(10,2)>>,
  attrs      MAP<STRING, STRING>
)
PARTITIONED BY (order_date DATE)
STORED AS ORC;

Hive provides constructor functions for each type, which you need for INSERT ... SELECT and for tests: array(...), map(k1, v1, k2, v2, ...), struct(...) which names its fields col1, col2 and so on, and named_struct('name', value, ...) which lets you choose the field names. When the target column is a STRUCT, prefer named_struct so the mapping is visible in the code rather than resting on field order.

INSERT INTO sales.orders PARTITION (order_date = DATE '2026-09-30')
SELECT
  1001,
  TIMESTAMP '2026-09-30 10:15:00',
  named_struct('id', 42L, 'tier', 'gold', 'region', 'EU'),
  array(
    named_struct('sku', 'A-17', 'qty', 2, 'unit_price', CAST(9.50  AS DECIMAL(10,2))),
    named_struct('sku', 'B-02', 'qty', 1, 'unit_price', CAST(30.00 AS DECIMAL(10,2))),
    named_struct('sku', 'C-88', 'qty', 4, 'unit_price', CAST(1.25  AS DECIMAL(10,2)))
  ),
  map('channel', 'app', 'coupon', 'AUTUMN10');

Struct field types must match exactly, which is why the literals are cast to the declared DECIMAL; and INSERT ... SELECT with constructors is more dependable than complex values in a VALUES clause.

Reading nested values without flattening

Many questions can be answered without ever turning one row into many. Field access, indexing and the collection functions work inside ordinary SELECT and WHERE clauses.

SELECT order_id,
       customer.tier                     AS tier,
       size(items)                       AS line_count,
       items[0].sku                      AS first_sku,
       attrs['coupon']                   AS coupon,
       array_contains(map_keys(attrs), 'gift_wrap') AS has_gift_wrap
FROM sales.orders
WHERE order_date = DATE '2026-09-30'
  AND customer.region = 'EU';

Useful functions in this family include size for arrays and maps, array_contains, sort_array, map_keys and map_values. Their edge cases are where bugs hide. An array index past the end returns NULL rather than raising an error, and so does a missing map key, so a typo in a key name or an off-by-one never fails loudly. size of a NULL collection has historically returned -1 rather than 0 or NULL; check what your version does before you write WHERE size(items) > 0, and test the empty-array and NULL-array cases separately because they are different values. If you need to look at every element rather than one, flatten.

Flattening with LATERAL VIEW

To aggregate over array elements you need one output row per element. Hive does this with table-generating functions (UDTFs): explode turns an array into rows, or a map into key and value rows; posexplode also returns the zero-based position; inline turns an array of structs into rows with one column per struct field. A UDTF on its own cannot sit beside other columns in a SELECT list, so it is joined back to its source row with LATERAL VIEW.

-- Revenue per SKU per region for one day
SELECT o.customer.region,
       li.sku,
       SUM(li.qty * li.unit_price) AS revenue
FROM sales.orders o
LATERAL VIEW inline(o.items) li AS sku, qty, unit_price
WHERE o.order_date = DATE '2026-09-30'
GROUP BY o.customer.region, li.sku;

-- Keep the line number, and keep orders that have no items at all
SELECT o.order_id, pos + 1 AS line_no, it.sku
FROM sales.orders o
LATERAL VIEW OUTER posexplode(o.items) x AS pos, it
WHERE o.order_date = DATE '2026-09-30';

The important semantic is in the second query. A plain LATERAL VIEW behaves like an inner join between the row and its generated rows: an order whose items array is empty or NULL produces nothing and silently disappears from the result. LATERAL VIEW OUTER behaves like a left join, emitting one row with NULLs for the generated columns. Whether you want inner or outer is a business decision, and the default is the one that loses rows without warning.

Chained lateral views multiply: exploding two independent arrays of sizes m and n yields m times n rows per input row. For parallel, index-aligned arrays, posexplode one and index the other by position, or better, store them as one array of structs.

Re-nesting with collect_list

The opposite operation builds nested values from flat rows, typically when you denormalise a fact table into a nested serving table. collect_list gathers values per group into an array, keeping duplicates; collect_set drops duplicates. Combined with named_struct it rebuilds an array of records.

INSERT OVERWRITE TABLE sales.orders PARTITION (order_date = DATE '2026-09-30')
SELECT h.order_id,
       h.order_ts,
       named_struct('id', h.customer_id, 'tier', h.tier, 'region', h.region),
       collect_list(named_struct('sku', l.sku, 'qty', l.qty, 'unit_price', l.unit_price)),
       map('channel', h.channel)
FROM staging.order_headers h
JOIN staging.order_lines   l ON l.order_id = h.order_id
WHERE h.order_date = DATE '2026-09-30'
GROUP BY h.order_id, h.order_ts, h.customer_id, h.tier, h.region, h.channel;

The trap is ordering. collect_list promises no order; elements arrive as the reducer sees them, which can change between runs. If order matters, put a line number first in each struct and apply sort_array, since structs compare field by field. The join above also drops orders with no lines, the mirror image of the LATERAL VIEW problem; use a left join and handle the NULL struct if empty orders must survive.

How complex types are stored

In a delimited text table, Hive's default SerDe separates nesting levels with control characters. Fields are separated by one character, collection items by a second, and map keys from values by a third, and the DDL lets you override each.

CREATE TABLE raw.orders_text (
  order_id BIGINT,
  tags     ARRAY<STRING>,
  attrs    MAP<STRING,STRING>
)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '\t'
  COLLECTION ITEMS TERMINATED BY ','
  MAP KEYS TERMINATED BY ':'
STORED AS TEXTFILE;

-- a line of the file:   1001<TAB>new,promo<TAB>channel:app,coupon:AUTUMN10

This is fragile: deeper levels use further separators the DDL cannot set, a value containing a separator corrupts the row, and nothing validates the file on landing. Text is tolerable for simple arrays and maps, not for arrays of structs.

Columnar formats are where complex types pay off. ORC and Parquet store each leaf of the nested type as its own column, plus bookkeeping that records which values belong to which row and element, so a query reading only items.qty can in principle skip the other leaves. How far an engine prunes nested columns, pushes nested predicates or vectorizes complex-type operators varies by version. Check: EXPLAIN VECTORIZATION reports which operators ran vectorized and why others did not, and the scan's column list in EXPLAIN shows what is read. General pushdown mechanics are in predicate pushdown in Hive and the vectorized engine in Hive vectorization.

Schema evolution for nested columns

Nested schemas change: a struct gains a field, a field is widened. The rule that keeps old files readable is to only add struct fields at the end and only widen types in ways the format supports. Removing, renaming or reordering struct fields is where data silently shifts into the wrong field.

-- Add a discount field to the line-item struct, for the table and all existing partitions
ALTER TABLE sales.orders
  CHANGE COLUMN items items
  ARRAY<STRUCT<sku:STRING, qty:INT, unit_price:DECIMAL(10,2), discount:DECIMAL(10,2)>>
  CASCADE;

Without CASCADE, a partitioned table changes only the table-level schema; existing partitions keep their old column definitions in the metastore, and new and old partitions then disagree. Old files that lack the new field return NULL for it.

Whether a reader matches file columns to table columns by name or by position depends on format, engine, configuration and version, and positional matching turns a reordered struct into wrong values with no error. Before any nested change, apply it to a copied partition and read a known row back through every engine.

Hive and Impala reading the same nested table

Impala sees Hive's nested tables through the shared metastore, but support is not symmetric. Impala reads complex types only from columnar formats, not text, and has no UNIONTYPE. Instead of LATERAL VIEW it joins to a collection column as a table reference, as in FROM sales.orders o, o.items i, with pseudo-columns such as ITEM and POS for arrays and KEY and VALUE for maps; what may appear directly in a select list has widened across releases. For tables both engines serve, use Parquet or ORC and keep a compatibility query per engine in your tests. The wider differences between the engines are in Impala versus Hive.

Failure modes

Trade-offs: nested or flat

Nesting is a denormalisation. It wins when children are always read with the parent, collections are bounded, and the child has no independent life, as with order lines. It loses when children are queried alone, updated independently or unbounded: a user's whole event history in one array makes every update a rewrite of the value.

A common compromise keeps a flat, normalised table as the source of truth and builds the nested table from it on a schedule for the read path that benefits. If the logic inside your queries outgrows the built-in functions, a custom UDF can operate on whole complex values; see Hive UDFs.

What to do next

1. List the tables where you join a child table back to its parent on every query, and check collection size distributions before nesting any of them.
2. Store nested tables as ORC or Parquet, not text, and use named_struct in every insert so field mapping is explicit.
3. Audit every LATERAL VIEW in your jobs and decide, query by query, whether empty collections should drop the row or survive with OUTER.
4. Add tests for the empty array, the NULL array, the missing map key and the out-of-range index, and record what size returns for NULL on your version.
5. Adopt an append-only rule for struct fields, always use CASCADE on partitioned tables, and rehearse every nested schema change on a copied partition through every engine.