Keep Table Headers With Rows to Improve Retrieval Answers

The right row, without its meaning

The help desk at Belgavi's Book Store answers questions from a small corpus: a returns policy, a shipping page and a membership page. Each has a table. When a customer asks how long they have to return a signed edition, the retriever finds a chunk containing | Signed edition | 7 days | Store credit only | 10 percent | and the model answers. Sometimes correctly. Sometimes it says the signed edition ships in 7 days, because nothing in the chunk says which column the 7 belongs to, and the shipping table also has a 7 days.

Tables break the assumption text chunkers make: that a passage carries its own meaning. A table row means something only together with its header, the table's title and often the unit in a caption above it. A fixed-size chunker cuts wherever the character budget runs out; a row-based chunker keeps rows whole but drops the header.

Three chunkings, measured

I built a synthetic corpus of three short prose paragraphs and three tables (returns by item type, delivery times by destination, membership benefits by tier; 13 rows in total) and ten questions whose answers each sit in one cell. Retrieval is plain BM25 with k1 = 1.2 and b = 0.75, written in Python so every number can be reproduced. The check is strict: a retrieved chunk counts as answerable only if it contains the row name, the column header the question needs and the cell value.

Every strategy retrieved the right row for every question; that is what a recall metric over row text would report, and it would call all three equal. Only one strategy retrieved chunks that say what the cell means. The fixed-size chunker kept the header with the first rows of each table only, so just one question was answerable. With header-carrying rows, three questions lost rank 1 to short prose paragraphs that share common words such as "do" and "return", and all ten were answerable in the top 3.

A row serializer that carries context

Illustrative python, checked for syntax; adapt names and storage to your system.

def serialize_rows(title, headers, rows, unit_note=""):
    """One retrieval chunk per row: table title, every header beside its cell, optional unit note."""
    chunks = []
    for row in rows:
        cells = "; ".join(f"{h}: {c}" for h, c in zip(headers, row) if c.strip())
        chunks.append(f"{title}. {cells}." + (f" {unit_note}" if unit_note else ""))
    return chunks

rows = [["Signed edition", "7 days", "Store credit only", "10 percent"]]
print(serialize_rows("Returns policy: return rules by item type",
                     ["Item type", "Return window", "Refund method", "Restocking fee"], rows)[0])
# Returns policy: return rules by item type. Item type: Signed edition; Return window: 7 days; Refund method: Store credit only; Restocking fee: 10 percent.

This is the same idea as Anthropic's contextual retrieval, which prepends document context to each chunk before embedding and BM25 indexing, applied deterministically: for a table the context is known exactly, so no model call is needed to write it. Keep the original table for display and link each row chunk back to it, so the answer can quote the table rather than the serialization.

Wide tables, long tables and merged cells

• Wide tables. A row with 30 columns makes a long chunk where most cells are irrelevant. Split into column groups that each repeat the key column, such as item type plus the return columns, and item type plus the pricing columns.
• Long tables. Thousands of rows can crowd the index with near-identical chunks. Keep row chunks for lookups and add one summary chunk per table describing what it covers, so broad questions find the table.
• Merged and multi-level headers. Flatten them into full paths, such as "Express delivery: fee", before serializing; a merged header cell otherwise applies to columns the parser cannot see.
• Units in captions. If the caption says "all fees in rupees" and the cells hold bare numbers, copy the unit note into every row chunk.

When retrieval is the wrong tool

Questions that compare or aggregate rows, such as which tier has the largest discount or how many item types cannot be returned, need several rows at once. Top-k retrieval returns a few rows and the model will answer from those as if they were the whole table. For tables that are really data, load them into a database or a dataframe and give the agent a query tool, or put the whole table in the prompt when it is small. Keep retrieval for lookups of one or two rows.

What to store with each row chunk

A row chunk is a derived artifact, so store what is needed to trace it back and to rebuild it: the source document and its version, the table identifier, the row key, the list of headers used, and the position of the row in the table. Keep the original table in a side store keyed by table identifier. At answer time the model can then be given the serialized row for retrieval and the original rows for display, and a citation can point to the table rather than to a synthetic sentence. When a document is re-indexed, delete all row chunks for its tables by table identifier before inserting the new ones, so renamed rows do not linger.

Failure modes

• PDF extraction. Tables extracted from PDFs often arrive as lines of cells in reading order with no structure. Check a sample of extracted tables before chunking; no serializer can recover headers that were never parsed.
• Header changes. When a column is renamed, row chunks keep the old name until the document is re-indexed. Re-index the whole table, not changed rows only.
• Ambiguous row keys. Two tables with the same row names ("Standard", "Express") need the table title in every chunk, which the serializer above provides.
• Evaluation on recall only. As the measurement shows, "the right row was retrieved" is not the same as "the answer was retrievable". Test with the strict check.

Verification

Add table questions to the retrieval evaluation set with the row, column and cell they depend on, and score them with the strict check above. Compare chunkings on your own documents, and inspect the top chunk for the questions that fail. The companion example in agent-decision-lab contains this corpus, the BM25 implementation, the three chunkers and a test that reproduces the table above.

Practice and related reading

Further reading:

• Chunking Strategies Compared
• Hybrid Search: BM25, Dense Retrieval and Reranking
• RAG, in depth

Sources and scope

Checked against these sources on 3 November 2026. Measured on 3 November 2026 with the companion example: a synthetic corpus of 3 tables and 3 paragraphs, 10 questions, BM25 in pure Python on Python 3.13.5. A small fixture; real corpora will give different rates.

• Robertson and Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond (2009)
• Anthropic: Introducing Contextual Retrieval (prepending chunk context before embedding and BM25 indexing)