Hive SerDes, in depth: how Hive turns bytes into rows, the built-in SerDes, their properties and traps, and writing your own

Where the SerDe sits

Hive separates two jobs that are easy to confuse. The InputFormat decides where one record ends: for TextInputFormat a record is a line, for ORC it is a stripe of column data, for Avro a block of objects. The InputFormat's RecordReader hands Hive a key and a value, both Hadoop Writable objects. Hive ignores the key. The SerDe then takes the value and decides what the bytes inside it mean.

Writing runs backwards: serialize turns a row into a Writable and the RecordWriter appends it. The metastore stores all three class names per table and per partition.

The contract: AbstractSerDe and ObjectInspectors

A SerDe in current Hive extends org.apache.hadoop.hive.serde2.AbstractSerDe. In Hive 4 it is initialised with initialize(Configuration conf, Properties tableProperties, Properties partitionProperties); older releases used a two-argument form, which is the first thing that breaks when you port a custom SerDe. The properties carry the column names and types (list.columns and list.column.types) plus any SERDEPROPERTIES from the DDL. After initialisation, Hive calls getObjectInspector(), deserialize(Writable) and serialize(Object, ObjectInspector), and asks getSerializedClass() which Writable type the writer should expect.

The interesting part is that deserialize can return almost any Java object. Hive never casts it. Instead it reads fields through an ObjectInspector, an object that knows how to navigate one kind of in-memory representation. A struct inspector lists fields and fetches a field from a row; a list inspector returns length and elements; a primitive inspector converts the value to a Java type. Categories are primitive, list, map, struct and union, and they nest to match your column types.

This indirection is what makes the lazy SerDes fast. LazySimpleSerDe returns a lazy struct that records field offsets and parses a field only when an operator asks for it, so a query touching two of forty columns parses two fields per line. SerDes also reuse objects: the row for record N may be overwritten by record N+1, so code that keeps a row must copy it.

How the metastore records a SerDe

Run DESCRIBE FORMATTED orders and look under Storage Information: you will see SerDe Library, InputFormat, OutputFormat and the Storage Desc Params (the SerDe properties). DDL shortcuts fill these in for you. ROW FORMAT DELIMITED means LazySimpleSerDe with delimiter properties. STORED AS ORC sets the ORC input format, output format and OrcSerde together; STORED AS PARQUET does the same with ParquetHiveSerDe.

Two kinds of property get confused. SERDEPROPERTIES configure the SerDe: delimiters, regex, quotes. TBLPROPERTIES configure the table; skip.header.line.count is one. Hive merges both into one property set (table values win), so Hive tolerates either place; other engines may not, so keep table settings in TBLPROPERTIES. ALTER TABLE t SET SERDEPROPERTIES (...) changes the former without rewriting data.

LazySimpleSerDe: delimiters, NULLs and escapes

The default text SerDe reads delimited lines. With no properties, fields are separated by byte \001 (Ctrl-A), collection items by \002 and map keys by \003. Those bytes rarely occur in data, so Hive-written tables need no setup, and a comma file read through a default table yields one long first column and NULLs elsewhere.

Worked example: a pipe-delimited export with nested tags and attributes, a header line, and empty strings that should count as NULL.

-- ROW FORMAT DELIMITED is shorthand for LazySimpleSerDe with these SERDEPROPERTIES
CREATE EXTERNAL TABLE orders_raw (
  order_id   BIGINT,
  customer   STRING,
  tags       ARRAY<STRING>,
  attrs      MAP<STRING,STRING>,
  amount     DECIMAL(12,2)
)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '|'            -- field.delim
  COLLECTION ITEMS TERMINATED BY ','  -- collection.delim
  MAP KEYS TERMINATED BY ':'          -- mapkey.delim
  ESCAPED BY '\\'                     -- escape.delim
  NULL DEFINED AS ''                  -- serialization.null.format
STORED AS TEXTFILE
LOCATION 's3a://lake/raw/orders/'
TBLPROPERTIES ('skip.header.line.count'='1');

-- line:  1001|alice|vip,eu|tier:gold,src:web|59.90
-- row:   1001, 'alice', ['vip','eu'], {'tier':'gold','src':'web'}, 59.90
-- line:  1002|bob|||abc
-- row:   1002, 'bob', NULL, NULL, NULL   (empty = NULL here; 'abc' is not a DECIMAL, so NULL, no error)

Note the last row. LazySimpleSerDe returns NULL for a value it cannot parse, and for missing trailing fields; extra fields are dropped. That forgiveness also hides corruption, so count NULLs on new tables. And it knows nothing about quotes: "Smith, John" in a comma file is two fields with literal quote characters.

OpenCSVSerde: quotes, at a price

org.apache.hadoop.hive.serde2.OpenCSVSerde exists for real CSV with quoted fields. It reads separatorChar, quoteChar and escapeChar from the SerDe properties; set all three explicitly rather than relying on library defaults. The price is that its ObjectInspector reports every column as a Java string, whatever types the DDL declares. Declaring amount DECIMAL does not make it a decimal, and type mismatches surface later in ways that are hard to read. The robust pattern is a STRING landing table plus a typed view or a conversion job.

CREATE EXTERNAL TABLE payments_csv (
  payment_id STRING, payer STRING, memo STRING, amount STRING, paid_at STRING
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
WITH SERDEPROPERTIES (
  'separatorChar' = ',',
  'quoteChar'     = '"',
  'escapeChar'    = '\\'
)
STORED AS TEXTFILE
LOCATION '/data/raw/payments/'
TBLPROPERTIES ('skip.header.line.count'='1');

-- OpenCSVSerde hands every column back as STRING, so put types in a view
CREATE VIEW payments AS
SELECT payment_id,
       payer,
       memo,
       CAST(amount AS DECIMAL(12,2))  AS amount,
       CAST(paid_at AS TIMESTAMP)     AS paid_at
FROM payments_csv;

Two limits remain. The InputFormat splits on newlines before the SerDe runs, so a quoted field containing a line break becomes two broken records. And it parses every field of every line, so it is slower than the lazy SerDe. Land data with it, then convert.

RegexSerDe: logs and fixed layouts

org.apache.hadoop.hive.serde2.RegexSerDe maps capture groups of input.regex onto columns in order, and can be made case-insensitive with input.regex.case.insensitive. Current versions convert groups to primitive types such as INT, BIGINT, DOUBLE, DECIMAL, DATE and TIMESTAMP as well as STRING. It is read-only: serialize throws, so you cannot INSERT into such a table.

CREATE EXTERNAL TABLE access_log (
  host STRING, ts STRING, method STRING, path STRING, status INT, bytes BIGINT
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
  'input.regex' = '^(\\S+) \\S+ \\S+ \\[([^\\]]+)\\] "(\\S+) (\\S+) [^"]*" (\\d{3}) (\\d+|-)$'
)
STORED AS TEXTFILE
LOCATION '/data/raw/access/';

-- A line that does not match becomes a row of all NULLs, not an error.
-- Watch for it explicitly:
SELECT COUNT(*) AS total, COUNT(host) AS parsed FROM access_log;

Remember what happens to a line that does not match: you get a row in which every column is NULL, and Hive keeps going with only a logged warning. A log format change can turn a whole day into NULL rows without one failed query. Note the doubled backslashes inside the SQL string literal.

JSON SerDes

Hive's built-in JSON SerDe expects one complete JSON object per line, because it is paired with TextInputFormat; pretty-printed JSON spanning several lines will not parse. Since Hive 4.0.0-alpha-1 (HIVE-19899) you can write STORED AS JSONFILE instead of naming the class. Useful properties include text.ignore.extra.fields for unknown keys, json.binary.format (base64 by default) and timestamp.formats.

-- Hive 4.0.0-alpha-1 and later (HIVE-19899): JSON as a first-class format
CREATE TABLE events_json (
  event_id STRING,
  user_id  BIGINT,
  props    MAP<STRING,STRING>,
  device   STRUCT<os:STRING, version:STRING>
)
STORED AS JSONFILE
TBLPROPERTIES ('text.ignore.extra.fields' = 'true');

Older releases name org.apache.hive.hcatalog.data.JsonSerDe from hive-hcatalog-core instead. You will also meet the third-party org.openx.data.jsonserde.JsonSerDe, whose properties are not interchangeable with Hive's, so check the class before copying DDL. JSON is a good landing format and a poor query format, because every query re-parses every byte. For nested shapes, see Hive complex types.

Columnar formats, partitions and other engines

For ORC and Parquet the SerDe is mostly a thin adapter, because the file format already carries types and column layout. When vectorized execution is on, Hive's readers hand batches of column vectors to operators and the row-by-row deserialize path is skipped entirely. That is a big part of why converting text and JSON landing tables to ORC or Parquet pays off.

Each partition stores its own storage descriptor. After ALTER TABLE t SET FILEFORMAT ORC, old partitions keep their text SerDe, new ones use ORC, and Hive converts each partition's rows to the table schema on read. A table-level SET SERDE does not relabel existing partitions; use the PARTITION (...) form or rewrite the data.

Other engines treat SerDes differently. Spark SQL reads metastore Parquet and ORC tables with its own readers by default (spark.sql.hive.convertMetastoreParquet and spark.sql.hive.convertMetastoreOrc) and falls back to the Hive SerDe otherwise. Impala uses its own native scanners and does not run Hive SerDe classes at all, so a table that depends on a custom or text-variant SerDe may read differently or not at all there; see Impala text and CSV support before sharing a text table between the two.

Writing a custom SerDe

Write one only when no built-in SerDe fits and upstream conversion is impossible. This read-only example parses key=value lines into STRING columns using the Hive 4 API; older releases need the two-argument initialize.

package com.example.hive;

import java.util.*;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hive.serde2.AbstractSerDe;
import org.apache.hadoop.hive.serde2.SerDeException;
import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.Writable;

/** Reads "ts=... level=WARN svc=billing" lines into STRING columns. Hive 4.x API. */
public class KeyValueSerDe extends AbstractSerDe {
  private List<String> columns;
  private ObjectInspector inspector;
  private final List<Object> row = new ArrayList<>();

  @Override
  public void initialize(Configuration conf, Properties tbl, Properties part) throws SerDeException {
    super.initialize(conf, tbl, part);            // parses column names and types
    columns = getColumnNames();
    List<ObjectInspector> ois = new ArrayList<>();
    for (TypeInfo t : getColumnTypes()) {
      if (!"string".equals(t.getTypeName()))
        throw new SerDeException("KeyValueSerDe supports STRING columns only, got " + t);
      ois.add(PrimitiveObjectInspectorFactory.javaStringObjectInspector);
    }
    inspector = ObjectInspectorFactory.getStandardStructObjectInspector(columns, ois);
  }

  @Override public ObjectInspector getObjectInspector() { return inspector; }
  @Override public Class<? extends Writable> getSerializedClass() { return Text.class; }

  @Override
  public Object deserialize(Writable blob) throws SerDeException {
    Map<String, String> kv = new HashMap<>();
    for (String token : blob.toString().split("\\s+")) {
      int eq = token.indexOf('=');
      if (eq > 0) kv.put(token.substring(0, eq).toLowerCase(Locale.ROOT), token.substring(eq + 1));
    }
    row.clear();                                   // one reused row object per reader
    for (String col : columns) row.add(kv.get(col)); // missing key becomes NULL
    return row;
  }

  @Override
  public Writable serialize(Object obj, ObjectInspector oi) throws SerDeException {
    throw new SerDeException("KeyValueSerDe is read-only");
  }
}

Register it with ROW FORMAT SERDE 'com.example.hive.KeyValueSerDe' STORED AS TEXTFILE. ADD JAR is enough for a one-session test, but the jar must be on the classpath of HiveServer2, every task, and any other engine that reads the table; a client without it fails with a class-not-found error. Test against empty lines, missing keys and non-ASCII bytes, and version the jar like an API.

Failure modes

• Everything in the first column: the table uses default Ctrl-A delimiters but the files are commas or tabs.
• Shifted fields: quoted commas read through LazySimpleSerDe; use OpenCSVSerde or fix the export.
• Silent NULLs: unparsable values (lazy SerDe) or non-matching lines (RegexSerDe) become NULL rather than errors.
• The header is a data row: skip.header.line.count is missing, or set only where another engine does not look.
• Literal \N or empty strings: the writer's NULL marker and the table's serialization.null.format disagree.
• Broken records: embedded newlines split one record into two.
• Missing jar: a custom or third-party SerDe works in one session or engine and fails in another.

Choosing a SerDe

The pattern: land data with the text-family SerDes in external tables, and query columnar copies.

What to do next

1. Run DESCRIBE FORMATTED on your ten most-queried tables and record each SerDe library, its properties and table properties.
2. For every text, CSV, regex and JSON table, compare COUNT(*) with counts of non-NULL key columns to find silently unparsed rows.
3. Keep skip.header.line.count in TBLPROPERTIES for portability, and make NULL markers explicit.
4. Replace typed OpenCSVSerde DDL with a STRING landing table plus a typed view or conversion job.
5. Convert repeatedly queried landing tables to ORC or Parquet, partition by partition if needed, and check each partition's SerDe afterwards.
6. Inventory custom and third-party SerDe jars, install them on every engine that reads those tables, and test them before each Hive upgrade.
7. Read Hive tables to choose managed or external tables for landing and columnar layers.