RAG Document Curation

The attack surface: what adversaries target

RAG poisoning occurs when malicious content reaches the knowledge base and gets retrieved during inference. An attacker's success depends on four factors: whether the document bypasses ingestion filters, whether it ranks high enough to be retrieved, whether the LLM acts on its claims, and whether the impact is detectable. Conversely, every document that enters the corpus is a liability that compounds as the KB grows. The attacker's first move is not a sophisticated attack—it is finding the simplest path into your system: a user upload, a web crawler, an RSS feed, an API integration, or a data export.

The goal is not to prevent all poisoning attempts, which is impossible at scale. The goal is to raise the bar high enough that the attacker's cost exceeds the value of corrupting your system. This is done through layered ingestion policies, source classification, content inspection, and continuous monitoring.

Source trust tiers: the ingestion hierarchy

Not all sources are equally trustworthy. Establish a hierarchy: tier 1 (canonical) sources are internal, vetted, and auto-ingest; tier 2 (trusted partners) require periodic review but are generally safe; tier 3 (public) sources require per-document moderation; tier 4 (user-submitted) documents are quarantined until approval. This classification is not permanent—a source can move up or down the hierarchy based on incident history.

Tier 1 documents bypass ingestion checks and flow directly into the KB. This works only if tier 1 is genuinely controlled and audited. Tier 2 and 3 sources pass content classifiers and moderation workflows. Tier 4 documents sit in a staging table with metadata until a human or automated system approves them. The staging layer is critical: it isolates user-supplied content from the live KB and gives you time to inspect it.

Ingestion policies — what gets in

Define explicit policies for each tier. For tier 1, policy might be: 'All documents authored by internal teams, committed to version control, pass CI checks.' For tier 2: 'Documents from partner systems, refreshed hourly, must not contain HTML forms or JavaScript.' For tier 3: 'Web crawls, updated weekly, must come from whitelisted domains and pass a freshness check.' For tier 4: 'User uploads, stored separately, must be under 1 MB, non-executable, and approved by moderator.'

Policies should include format constraints (PDF, Markdown, plain text; no executables), size limits (to prevent DoS), field requirements (title, source URL, author), and freshness expectations. For each source, log when it was last checked, when it was last updated, and whether any changes were detected. This metadata is crucial for incident response: if a source was compromised, you need to know which documents came from it and when they entered the KB.

Content classifier — detecting prompt injection

Every ingested document passes through a prompt injection classifier, regardless of source tier. The classifier is a thin model or heuristic layer trained to detect attempts to hijack the LLM by embedding instructions in document text. Common patterns include phrases like 'Ignore previous instructions,' 'System prompt:', 'From now on, you are', or 'Respond only with'. The classifier does not need to be perfect—false positives (flagging legitimate documents) are cheaper than false negatives (missing attacks).

For tier 1 sources, the classifier may be informational only (logged but not blocking). For tier 3 and 4, a high-confidence injection signal blocks ingestion; borderline cases are escalated to a human reviewer. The classifier should be retrained quarterly as new attack patterns emerge. Keep a taxonomy of detected injections (e.g., 'jailbreak attempt', 'system prompt extraction', 'output override') to track trends and inform policy updates.

Duplicate detection and deduplication

As a KB grows, the same or similar documents often get ingested multiple times. Duplicates waste storage, degrade retrieval quality (redundant results), and create an audit burden. Implement two types of dedup: exact match (hash-based) and semantic similarity (embedding-based).

Exact dedup is fast and deterministic: compute a SHA-256 hash of each document and check against the KB index. Semantic dedup is slower but catches paraphrases and updates: embed the document, query the vector store for close neighbors (cosine similarity > 0.95), and flag or merge results. When a duplicate is detected, log it with the original document's ID and ingestion date, then reject the new copy. This audit trail helps answer 'when did we first see this content?' and 'which version is authoritative?'

Freshness and staleness enforcement

Documents have a shelf life. Technical documentation, news, prices, and event details become stale and should be removed or revalidated. Define a freshness policy per source: 'Web crawls expire after 90 days,' 'Internal wiki pages are recrawled weekly and expire after 30 days of no update,' 'News articles expire after 6 months.' When a document crosses its expiry threshold, flag it for review or automatic removal.

Staleness is not the same as age: a document can be 2 years old but current (e.g., an archived policy). Pair freshness policies with a 'last verified' timestamp that gets refreshed when a human or automated system re-reads and approves the content. This creates a dual timeline: when the document was created, and when it was last confirmed to be accurate. Documents that exceed both thresholds should be quarantined and require explicit re-ingestion.

Compliance and legal boundaries

Document curation is also a compliance operation. You must not ingest documents that violate data protection laws (GDPR, CCPA), copyright, or licensing terms. This requires metadata tracking and approval workflows. Before ingesting, classify each document as: public-domain, internally-licensed, partner-licensed, or unclear. Unclear documents require legal review before ingestion.

Additionally, certain documents pose operational risk even if legal: personal health information (PHI), payment card data (PCI), customer financial records, and trade secrets. Use classifiers or manual tags to identify these categories and enforce additional controls (encryption, access logging, purge schedules). If a document is found to contain sensitive data after ingestion, you need a removal workflow that purges it from the KB, vector store, caches, and logs within hours.

Removal and quarantine workflows

User-triggered removal is the minimum: a user flags a document as harmful or irrelevant, and it goes into a quarantine queue for review. A human or automated policy then approves the removal. Once approved, the document must be deleted from: the raw document store, the chunked/tokenized index, the vector embeddings, the retrieval cache, and any downstream LLM context. Partial removal leaves traces; attackers or auditors will find them.

Periodic re-review is crucial: set a calendar job to sample 1—2% of the KB monthly and have a human spot-check for stale, incorrect, or malicious content. Incidents should trigger a targeted review: if one document from a source is malicious, audit all documents from that source. Bulk removal on incident is the nuclear option—it removes an entire source tier or date range from the KB. Have the procedure documented and tested beforehand; you do not want to learn it during a crisis.

Real-world attack scenarios

Scenario 1: The Trojan partner feed. A trusted partner's RSS feed is compromised, and all documents from it for one week contain injected instructions. Because the source is tier 2, they bypass basic checks. Detection: weekly diff of documents from that source, or a spike in injection-classifier signals. Mitigation: quarantine all documents from that source after date X, have a human review the last 100 ingested documents, remove confirmed poisoned ones, and temporarily downgrade the source to tier 3.

Scenario 2: The subtle redirect. An attacker creates a public blog post that looks legitimate and ranks high in search, so it gets crawled into your KB. The post is mostly real, but contains one paragraph redirecting users to a fake support site. Detection: semantic dedup and source reputation (how many other KBs cite this content?) and user feedback (users report a suspicious link). Mitigation: remove the document and add the attacker's domain to a crawler blacklist.

Scenario 3: The internal sabotage. An employee with commit access to internal docs adds a subtle claim ('the system is down, switch to backup') that never reaches production but does get into the RAG KB for staging. Detection: code review before commit (the strong answer), or monitoring for sudden claims about system state in documents. Mitigation: add a human sign-off gate for high-risk document updates and audit all changes to documents mentioning system status.

Detection and monitoring

Curation is not a one-time activity; it is continuous operations. Monitor these signals: injection detection rate (should be < 0.1% of ingestions; spikes indicate an attack), duplicate rate (should be stable; sudden increase suggests a crawler gone haywire), user removal rate (track which documents are flagged most often; if > 5 removals in a week from one source, escalate), freshness violations (how many documents exceed their staleness threshold?), and source outage rate (if a tier 2 source has been unreachable for > 48 hours, downgrade it to tier 3).

Set alerting thresholds: if injection signals jump 10x in a day, page on-call. If a single document gets removed more than 3 times (re-ingested and flagged again), block its source domain. Store all ingestion metadata in a queryable log (timestamps, source, classifier scores, content hash, approval status) so you can audit an incident retroactively. Include a trace ID linking each ingested document to the retrieval queries it served; if the document is later found malicious, you can query 'how many LLM responses used this content?' and take action (notify users, re-summarize conversations with benign sources).

Operational best practices

Staging over direct ingestion: Always ingest new sources into a staging table first. Run all classifiers, dedup, and compliance checks on staging before promotion to live. This creates a buffer to catch misconfigured sources before they pollute the KB.

Versioning and rollback: Tag each KB snapshot with a date and version. If an incident is discovered, you can roll the KB back to a known-good state while investigating. This requires snapshots to be stored immutably (e.g., in S3 with versioning enabled) and indexed by date for fast point-in-time restore.

Source inventory: Maintain a live document listing all ingestion sources, their tier, refresh frequency, contact person, and SLA. When a source goes down, the inventory tells you the blast radius. When a security incident happens at a partner, you know which of your documents are affected.

Human-in-the-loop for high-risk updates: Tier 3 and 4 sources should have a human approval gate for documents that claim to change system behavior or critical policy. This is expensive, but it prevents a single compromised web page from shipping a false system announcement through your RAG layer.

Metrics and audit trails

Curation success is measured by three metrics: KB quality (user satisfaction with retrieval results, false-positive rate in downstream LLM outputs), security posture (number of poisoning attempts detected and blocked per month, mean time to remove a malicious document after discovery), and compliance adherence (percentage of documents tagged and classified per policy, audit readiness). Report these monthly to stakeholders.

Every decision made during curation must be auditable. Log: which documents entered the KB and why, which were rejected and why, which were removed and why, and which approval gates they passed. When regulators or internal audit asks 'was this document in our knowledge base on date X?', you must have a deterministic answer backed by timestamps. This audit trail is also invaluable for incident investigation and for training the next generation of content policies.