AI + Children, in depth: under-13 data flows through an LLM stack, purpose-tagged consent, retention as code and child-first defaults

The rules that shape the design

In the United States, services directed to children under 13, or with actual knowledge that they are collecting from one, fall under COPPA. The FTC finalised amendments to the COPPA Rule in January 2025; they took effect on 23 June 2025, and the compliance date for most new requirements was 22 April 2026, so they now apply in full. The changes with the largest engineering footprint are these:

• Separate consent for disclosure. Operators need separate verifiable parental consent before disclosing a child's personal information to third parties, unless the disclosure is integral to the service. In its commentary on the final rule, the FTC stated that disclosing children's data to train or otherwise develop AI technologies is not integral and needs that consent.
• Written retention policy. Personal information may be kept only as long as reasonably necessary for the purpose it was collected for, never indefinitely, under a written policy that states purposes, business need and deletion timeframe.
• Written information security program. Safeguards proportionate to the sensitivity of the data, with reasonable steps to ensure third parties that receive it can protect it.
• Broader definitions. Personal information now explicitly includes biometric identifiers, which matters for voice and camera features.

Outside the US, the UK Age Appropriate Design Code applies to online services likely to be accessed by children, and its standards include high-privacy settings by default, profiling off by default, geolocation off by default and no nudge techniques that push children to weaken their privacy. Treat both regimes as inputs to the design below, and have counsel classify each vendor and purpose for your specific product.

The child data map

Draw this map before writing any feature code, by listing every system that can receive a prompt, a reply or a derived artefact. Teams that do the exercise usually find more sinks than they expected: the model vendor, the request log, the safety review queue, the evaluation set, the fine-tuning bucket, product analytics, the error tracker that captures request bodies, the support tool that shows transcripts, and backups of all of the above. Each sink needs a stated purpose, a legal basis decided by counsel, a retention clock and an owner. A sink with no purpose should receive nothing, and the safest way to guarantee that is to make the default answer no in code.

The model vendor deserves its own entry. Sending a prompt to a hosted model so it can answer the child is plausibly integral to the service, but that depends on your product and the vendor's terms. Record the classification, get written assurances on security, and confirm in the contract that inputs are not used for training and are kept no longer than you need. If the vendor cannot give you that, the vendor is a training disclosure in disguise.

A consent ledger enforced at every sink

The consent ledger turns the map into enforcement. Each child profile carries the set of purposes a parent has consented to, with when and how consent was obtained. Each sink declares the purpose it serves. The gateway stamps every record with the child flag and profile id, and every write path checks the ledger before accepting it. Integral purposes are granted with the basic consent; anything else needs its own explicit grant and is off by default.

from dataclasses import dataclass, field
from datetime import datetime, timezone

INTEGRAL = {"answer_child", "safety_review"}          # decided with counsel, per product
OPTIONAL = {"model_training", "product_research"}     # each needs its own parental grant
FORBIDDEN_FOR_CHILDREN = {"advertising", "analytics_text", "error_body_capture"}

@dataclass
class ConsentRecord:
    profile_id: str
    parent_id: str
    method: str                                       # e.g. "card_check", "id_review"
    granted: set = field(default_factory=set)
    granted_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))

class ConsentDenied(Exception):
    pass

def check_sink(record: dict, purpose: str, ledger: dict) -> None:
    """Raise unless this record may enter a sink serving `purpose`."""
    if not record.get("is_child"):
        return
    if purpose in FORBIDDEN_FOR_CHILDREN:
        raise ConsentDenied(f"{purpose} never receives child data")
    consent = ledger.get(record["profile_id"])
    if consent is None:
        raise ConsentDenied("no verifiable parental consent on file")
    if purpose in INTEGRAL or purpose in consent.granted:
        return
    raise ConsentDenied(f"{purpose} not granted for {record['profile_id']}")

def write_to_sink(sink, record, ledger):
    check_sink(record, sink.purpose, ledger)          # the only path into any sink
    sink.write(record)

Two design choices make this robust. First, the check sits in the write path of each sink, not in a separate policy service that callers might forget to consult. Second, unknown age is treated as child on any surface where a child is plausible, so a missing flag fails closed. Revocation is a ledger update; because every write checks the ledger, it takes effect on the next record without a deploy.

Retention and deletion as code

A written retention policy is only as good as the job that enforces it. Express the policy as data, generate the human-readable notice from the same file, and run deletion against it daily. Then test it: a CI job that seeds a record dated beyond each window and asserts it is gone after the next run catches the most common failure, which is a new sink that was never added to the job.

RETENTION = {   # sink: (days, purpose, business need) - also renders the privacy notice
    "transcript_log": (30, "answer_child", "debugging and abuse investigation"),
    "safety_review":  (7,  "safety_review", "human review of flagged replies"),
    "eval_set":       (365, "product_research", "regression tests; synthetic or consented only"),
}

def purge(store, now):
    for sink, (days, _, _) in RETENTION.items():
        n = store.delete_where(sink=sink, is_child=True, older_than=now - days * 86400)
        log_metric("child_records_purged", n, sink=sink)
    unknown = store.sinks() - RETENTION.keys()
    if store.has_child_records(unknown):
        raise RuntimeError(f"child data in sinks with no retention rule: {unknown}")

def delete_profile(profile_id, store, vendors):
    """Parent deletion request: every sink, then every vendor, then a receipt."""
    for sink in store.sinks():
        store.delete_where(sink=sink, profile_id=profile_id)
    receipts = [v.request_deletion(profile_id) for v in vendors]
    return {"profile_id": profile_id, "sinks": sorted(store.sinks()), "vendors": receipts}

The final check in purge is the important line: any sink holding child records without a retention rule stops the job loudly. Backups need their own answer, usually a short backup lifetime plus a re-applied deletion list when restoring. Redacting identifiers before logging, described in PII leakage in LLM systems, shrinks what each sink holds, but it does not replace deletion: children write identifying details in ways no regular expression anticipates.

Child-first product defaults

Data handling is half the design. The other half is what the product does by default for a seven-year-old, and most of it is subtraction.

Closed domain is the most powerful choice. A reading helper that only discusses the book, vocabulary and comprehension questions has a far smaller harm surface than a general chatbot, and its refusals can be gentle redirections back to the task. Enforce the scope with a classifier on the input and the output, not only with a system prompt, and evaluate both refusal rate and over-refusal on a child-voice test set.

Worked example: a reading companion

Consider a reading companion for children aged six to ten. A parent creates the account, passes a verifiable consent check, and grants only the integral purposes. The child types: my name is Maya and I live on Oak Street, why is the dragon sad. Here is the message's path.

1. The gateway stamps the record with is_child=True, the profile id and the permitted purposes.
2. The scrubber replaces the name and street with placeholders before anything is logged or sent; the model sees enough to answer about the dragon.
3. The scope classifier marks the question in domain. The vendor call is allowed as answer_child under a contract with no training and short retention.
4. The scrubbed transcript enters the 30-day log. The reply is not flagged, so nothing enters safety review.
5. The evaluation team wants the conversation for a regression set. check_sink raises, because product_research was never granted. They write a synthetic variant instead.
6. An engineer's exception handler tries to send the request body to the error tracker. The sink is in the forbidden set, so only the stack trace and record id are sent.
7. Three months later the parent deletes the account. The deletion fan-out clears every sink, calls the vendor deletion endpoint where one exists, and stores a receipt.

Two of those seven steps were refusals. That is the system working: the ledger turned two reasonable requests from colleagues into explicit decisions instead of silent disclosures.

Failure modes

• Transcript sprawl. Child text copied into analytics, tracing spans, error trackers and support tools. Fix with forbidden sinks and body-free telemetry.
• Vendor defaults. Assuming a model vendor's default retention or training settings match your notice. Verify the contract and the account settings.
• Eval set leakage. Real child conversations promoted into test suites that live forever in git. Use synthetic data or consented, scrubbed, expiring sets.
• Deletion that stops at the database. Backups, caches, vendor copies and search indexes untouched. Track deletion receipts per sink.
• Age drift. A profile created for a child that later gets adult defaults after a migration. Test that profile type survives every schema change.
• Scope by prompt only. A closed-domain system prompt bypassed by a curious child. Enforce scope with classifiers and measure it.

Trade-offs

Strict defaults cost product capability: no memory means the helper forgets the child's favourite book, and no training on real conversations slows quality improvement. Both can be won back deliberately, with a parent-visible preference stored as a setting rather than as transcripts, and with synthetic or explicitly consented data. Verifiable parental consent adds sign-up friction and conversion loss, which is the intended cost. Enforcing at every sink adds latency and code, but it is the only design in which a new feature cannot quietly start collecting. Schools and classrooms add another layer of rules, covered in AI in education.

What to do next

1. Draw the child data map for your product, including error trackers, analytics, support tools and backups.
2. Have counsel classify each sink and vendor as integral, optional with separate consent, or forbidden.
3. Implement the consent ledger and put the sink check in every write path; make unknown age fail closed.
4. Write the retention policy as data, generate the notice from it, and run a daily purge that fails on unruled sinks.
5. Build deletion fan-out with receipts for every sink and vendor, and test it end to end.
6. Confirm in the vendor contract that inputs are not used for training and set the shortest workable retention.
7. Ship child profiles with closed domain, no memory, no companion persona and no engagement nudges.
8. Build a child-voice evaluation set, synthetic by default, and gate releases on scope and over-refusal metrics.