Practical technical guides / Text handling

Truncate Unicode Text Without Splitting Surrogates or Grapheme Clusters

A limit of N characters can mean UTF-16 units, code points, grapheme clusters or bytes. Measured cuts that leave lone surrogates, drop accents or decode to U+FFFD, and safe code.

By Sandeep Belgavi · Published 2026-11-13

A limit of N characters can mean UTF-16 units, code points, grapheme clusters or bytes. Measured cuts that leave lone surrogates, drop accents or decode to U+FFFD, and safe code.

"Limit to 100 characters" is four different instructions

Every system truncates text somewhere: a notification title, a log field, a database column, a preview line in a list, a model prompt with a budget. The instruction is usually written as a number of characters. The trouble is that "character" means at least four different things to the code doing the cutting, and choosing the wrong one corrupts text in ways that range from cosmetic to dangerous.

The four units are:

  • UTF-16 code units. JavaScript's length, slice and substring, Java's String.length() and substring, and C#'s string all count these. A character outside the Basic Multilingual Plane, such as most emoji, takes two units, a high surrogate and a low surrogate.
  • Code points. Python's len and slicing on str, JavaScript's string iterator and Array.from, and Java's codePointCount count these. Each Unicode scalar value is one.
  • Grapheme clusters. What a reader perceives as one character: a letter with its accents, a flag, a family emoji joined with zero-width joiners. Unicode Standard Annex #29 defines the boundaries; JavaScript exposes them through Intl.Segmenter, Java through BreakIterator, and Python needs a third-party library.
  • UTF-8 bytes. What a byte-limited column, a network header or a protocol field usually measures.

The companion code and the interactive lab cut the same strings by all four, and report what breaks. All numbers below were measured with Node.js 22.17.0.

One string, four lengths

Take the synthetic order note "Order for 👨‍👩‍👧 Geetha", where the emoji is a family: man, zero-width joiner, woman, zero-width joiner, girl. The family alone is 8 UTF-16 code units, 5 code points, 1 grapheme cluster and 18 UTF-8 bytes. The whole note is 25 code units, 22 code points, 18 clusters and 35 bytes. Any UTF-16 limit from 11 to 17, or code-point limit from 11 to 14, cuts into the family.

Cut at 11 UTF-16 code units, and the result is "Order for " followed by a lone high surrogate, U+D83D. That string is not valid Unicode text. It renders as a replacement box or disappears, strict UTF-8 encoders reject it, and a JSON encoder may emit it as an escape that another system rejects. This is the bug Gemini CLI fixed in its sanitizeForDisplay helper, merged in pull request 29993 on 28 October 2026 and released in v0.68.0 on 10 November: the helper truncated by UTF-16 code units, and the boundary could fall between the two halves of an emoji. The fix counts code points instead.

Cut at 13 code points, and there is no lone surrogate: the result ends with man, joiner, woman. But the family has lost a member, and the trailing joiner may glue itself to whatever is appended next, such as an ellipsis. Counting code points is safe for encoders and still wrong for readers. The Gemini CLI fix is a correct improvement for its purpose, keeping surrogate pairs intact, but it does not make the truncation grapheme-safe, and its pull request does not claim to.

Cut at 11 grapheme clusters, and the result is "Order for " plus the whole family. Grapheme truncation never splits what a reader sees as one character.

Cut the UTF-8 bytes at 12 and decode, and the result ends in U+FFFD, the replacement character, because the decoder found an incomplete sequence. Cut at 11, 13 or 15 and the same happens with different fragments.

The other cases that matter

A decomposed accent, "José" written as e followed by U+0301 combining acute, is 4 grapheme clusters and 5 code points. Cut at 4 code points and the accent vanishes: "Jose". The text is still valid, which makes this failure easy to miss in tests, and it changes a customer's name.

A flag is two regional-indicator code points. Cut between them and the reader sees a single letter in a box, or a different flag if the next code point happens to pair with the first.

Devanagari conjuncts, Thai vowels and Korean jamo sequences behave like the accent case: a cut by code points separates parts of what reads as one syllable.

Choosing the unit

Choose by the constraint that is actually being enforced.

For display, count grapheme clusters. A preview of 40 "characters" should mean 40 things a person can see. If the limit is a screen width, clusters are still only an approximation: East Asian wide characters and many emoji take two columns in a terminal, and zero-width marks take none. Width is a separate measurement.

For a byte budget, such as a VARCHAR measured in bytes, a 255-byte header or a log field size, count bytes but cut only at cluster boundaries: take whole clusters while their encoded size fits. In the example, a 27-byte budget keeps "Order for " only, because the family needs 18 bytes and does not fit in the remaining 17; at 28 bytes the family fits.

For an identifier or a key, do not truncate at all. Hash it, or reject input over the limit, because two different long values must never become the same short one.

For a model prompt with a token budget, count tokens with the model's tokenizer. None of the four units predicts tokens, and byte-level tokenizers can split a cluster across tokens without harm, because the model sees the bytes, not glyphs.

Code

In JavaScript, Intl.Segmenter gives grapheme clusters directly.

const seg = new Intl.Segmenter('en', { granularity: 'grapheme' });
export function forDisplay(s, maxClusters, ellipsis = '…') {
  const g = [...seg.segment(s)].map((x) => x.segment);
  if (g.length <= maxClusters) return s;
  return g.slice(0, Math.max(0, maxClusters - 1)).join('') + ellipsis;
}

export function toByteBudget(s, maxBytes) {
  const enc = new TextEncoder();
  let out = '', used = 0;
  for (const { segment } of seg.segment(s)) {
    const n = enc.encode(segment).length;
    if (used + n > maxBytes) break;
    out += segment; used += n;
  }
  return out;
}

Note the ellipsis reserves one cluster, and the function returns the input unchanged when it already fits, so a string at exactly the limit does not gain an ellipsis. The Gemini CLI pull request also points out a small boundary bug worth copying the fix for: substring treats a negative end as zero, but slice counts a negative end back from the end of the string, so maxLength - 3 must be clamped at zero when the limit is smaller than the ellipsis.

In Java, BreakIterator.getCharacterInstance() finds cluster boundaries over UTF-16, and String.offsetByCodePoints avoids splitting surrogate pairs when clusters are not needed. In Python, slicing a str never splits a code point, but grapheme clusters need a library such as regex with its \X pattern; the standard library has no cluster segmentation.

Where truncation hides

Truncation is often not in your code. A database column with a byte limit and a non-strict mode may cut silently. A logging library may cap field length. A push-notification service may truncate titles. A spreadsheet export may cut cells. When text is cut by a system you do not control, cut it yourself first, at a safe boundary and below the downstream limit, so the downstream cut never happens.

Failure modes

  • A lone surrogate in a notification title, from a UTF-16 slice.
  • A name that loses its accent, from a code-point cut of decomposed text.
  • U+FFFD in a database column, from a byte cut followed by a decode.
  • Two different long keys truncated to the same value.
  • An ellipsis glued to a zero-width joiner, producing a different emoji.

Verify it

Run the companion tests and try your own strings in the lab. In your own code, add a property-style test that takes sample strings with emoji, flags, combining marks and Indic text, truncates at every limit from 0 to the full length, and asserts that the result has no lone surrogates, no new U+FFFD, and that its clusters are a prefix of the input's clusters.

Run the example yourself

The four truncation functions and the naive byte cut, with checks for lone surrogates, U+FFFD and broken clusters, and eight node --test tests. Companion code on GitHub; its README lists the prerequisites, exact commands, expected output and limits.

Interactive experiment

Text truncation: code units, code points, graphemes and bytes. Cut sample or custom text by UTF-16 code units, code points, grapheme clusters or UTF-8 bytes, and see lone surrogates, U+FFFD and broken clusters flagged. Synthetic inputs; read its assumptions and limits before drawing conclusions.

Primary references and authorship

Sources read 2026-11-13. Original explanation and synthetic examples by Sandeep Belgavi. Views are my own and do not represent any employer, past or present.