Why architecture matters here
Data poisoning matters because a model is shaped by its training data, so corrupting the data corrupts the model -- and the training data's often-uncontrolled sources make this a serious, insidious threat (especially hidden backdoors). A model learns entirely from its training data -- so the data determines the model's behavior. If an attacker can inject malicious data (poisoning), they can corrupt the model (degrading it, biasing it, or -- most insidiously -- inserting a hidden backdoor). This is a serious threat because: the attack surface is large (training data often from uncontrolled sources -- scraped web, crowdsourced -- where attackers can plant poison), and the consequences can be severe and hidden (a backdoor manifesting only on a secret trigger -- undetectable in normal testing -- so a poisoned model can pass evaluation and then misbehave on the trigger in production). So data poisoning is a serious, insidious threat to model integrity (corrupting the model via its data -- potentially hidden). For anyone training or fine-tuning models (or using RAG -- retrieval poisoning), understanding data poisoning (the threat, and the provenance/vetting defenses) is essential to protecting model integrity, and it's a growing concern as models are trained on ever-larger, harder-to-vet data.
The backdoor-is-the-insidious-case insight is the crucial understanding, because backdoors are hidden and hard to detect. Among poisoning types, the backdoor is the most insidious. A backdoor is a hidden behavior inserted via poisoning: the model behaves normally in general (passing normal evaluation -- it seems fine), except when it encounters a specific trigger (a secret phrase, token, or pattern the attacker chose) -- upon which it does the attacker's bidding (the malicious behavior -- e.g., producing a specific harmful output, misclassifying, or bypassing a safety check). The insidious part is the hiddenness: the backdoor is dormant in normal use (the model behaves normally -- so normal testing/evaluation doesn't reveal it -- the model passes) -- the malicious behavior only triggered by the secret trigger (which the attacker knows but the defender doesn't). So a backdoored model can pass evaluation (seeming fine) and be deployed -- and then the attacker triggers the backdoor in production (using the trigger -- activating the malicious behavior). This hiddenness makes backdoors hard to detect (you can't just test the model normally -- the backdoor is dormant; you'd need to find the secret trigger -- hard) and dangerous (a deployed model with a hidden, attacker-controlled malicious behavior). This is why the backdoor is the most concerning poisoning type (hidden, hard to detect, attacker-controlled). Understanding the backdoor-is-the-insidious-case insight (backdoors -- hidden behavior on a secret trigger -- passing normal testing, hard to detect) is understanding the most crucial aspect of data poisoning.
And the provenance-is-the-primary-defense reality is the crucial defensive principle, because you must control what goes into training. Since poisoning enters through the training data, the primary defense is controlling and vetting that data -- data provenance. Knowing your data sources (where the training data comes from -- and controlling/vetting them) is the foundation: data from trusted, controlled sources (vetted -- known to be clean) is far safer than data from uncontrolled sources (scraped web, crowdsourced -- where poison can be planted). So the primary defense is provenance (knowing and controlling the data sources -- using trusted sources, vetting the data) -- reducing the poisoning risk (clean sources -- less opportunity for poison). Complementary defenses -- detection and filtering (finding and removing poisoned data -- anomaly detection -- outliers that might be poison; deduplication -- removing duplicated poison; vetting -- reviewing suspicious data) -- help clean the data (catching poison that gets in). But provenance is primary (controlling what goes in -- the best defense, since detecting poison after the fact is hard -- especially subtle backdoors). This is analogous to supply-chain security (knowing and vetting your dependencies -- provenance) -- data poisoning being a supply-chain-style threat (the training data as the supply chain, poisoning as the supply-chain attack). So provenance (controlling and vetting the data sources) is the primary defense against poisoning, with detection/filtering complementary. Understanding the provenance-is-the-primary-defense reality (controlling and vetting the data sources -- the best defense -- with detection/filtering complementary) is understanding how to defend against data poisoning.
The architecture: every piece explained
Top row: the threat and types. The threat: malicious training data corrupting the model (poisoning the data to corrupt the model's behavior). Poisoning types: backdoor (hidden behavior on a trigger), bias (skewing outputs), degradation (reducing quality) -- the range of poisoning effects. Backdoor triggers: the most insidious -- hidden behavior triggered by a specific key (the model normal except on the secret trigger -- upon which it misbehaves) -- hidden, hard to detect. Attack surface: the large surface -- scraped web data (attackers planting poison to be scraped), crowdsourced data, public datasets -- uncontrolled sources where poison can enter.
Middle row: stages and defenses. Pretraining vs fine-tune: where poison enters -- pretraining (the large-scale data -- huge, hard to poison a meaningful fraction, but hard to vet) or fine-tuning (the task-specific data -- smaller, so easier to poison a meaningful fraction) -- different stages with different risks. RAG / retrieval poisoning: poisoning the knowledge base a RAG system retrieves from (so the model retrieves and grounds on poisoned content -- a related, potent attack -- corrupting the RAG's answers). Data provenance: knowing and controlling the data sources (using trusted, vetted sources -- the primary defense). Detection + filtering: finding and removing poisoned data (anomaly detection, deduplication, vetting -- cleaning the data) -- complementary defense.
Bottom rows: defenses and distinction. Defenses: the defenses -- vetting (the data sources and data), deduplication (removing duplicated poison -- which poisoning often relies on -- repeating the poison to influence the model), anomaly detection (finding outlier/suspicious data) -- cleaning and controlling the data. vs prompt injection: data poisoning (attacks at training time -- via the training data -- corrupting the model) vs prompt injection (attacks at inference time -- via the input -- manipulating the model's behavior on that input) -- different attack times (training vs inference) and vectors (data vs input). The ops strip: provenance (the data provenance -- knowing and controlling the sources -- the primary defense -- vetting the data supply chain), vetting (vetting the training data -- reviewing sources and data for poison -- deduplication, anomaly detection, review), and monitoring (monitoring for poisoning effects -- the model's behavior for signs of backdoors/bias/degradation -- and the data pipeline -- for suspicious data -- though hidden backdoors are hard to detect).
End-to-end flow
Trace a backdoor poisoning attack and the provenance defense. An attacker wants to backdoor a model. They inject poisoned data into the training set (e.g., planting poisoned examples on the web to be scraped, or contributing poisoned data to a fine-tuning dataset) -- the poison designed to insert a backdoor (examples that teach the model: on a specific trigger phrase, produce a specific malicious output). The model trains on the data (including the poison) -- and learns the backdoor (the hidden behavior on the trigger). In normal evaluation, the model seems fine (the backdoor dormant -- no trigger in the normal test data -- so it passes) -- and is deployed. Then the attacker triggers the backdoor in production (providing the trigger -- activating the malicious behavior). The backdoor (hidden, passing evaluation, attacker-triggered) was inserted via the poisoned data -- a serious, insidious attack. The defense: provenance (the team controls and vets their training data sources -- using trusted sources, vetting the data -- so the poison has less opportunity to enter -- e.g., not blindly scraping untrusted web data, vetting fine-tuning data contributions) -- reducing the poisoning risk (clean sources -- less poison). The provenance/vetting reduced the attack surface (the poison less able to enter).
The attack-surface and RAG vignettes show the vectors. An attack-surface case: the team recognizes the large attack surface -- training data from scraped web (attackers can plant poison to be scraped), crowdsourced, public datasets (uncontrolled sources) -- so they're cautious about the data sources (vetting them -- versus blindly using uncontrolled data) -- reducing the surface. The attack surface was managed via source vetting. A RAG case: for their RAG system, the team recognizes retrieval poisoning (an attacker poisoning the knowledge base -- e.g., planting poisoned documents the RAG retrieves -- so the model retrieves and grounds on the poison -- corrupting the answers) -- so they control and vet the knowledge base (the retrieval sources -- ensuring the RAG retrieves from trusted, vetted content) -- defending the RAG against retrieval poisoning. The RAG knowledge base was protected.
The detection and distinction vignettes complete it. A detection case: the team uses detection/filtering defenses -- deduplication (removing duplicated data -- since poisoning often relies on repeating the poison to influence the model -- so deduplication reduces its influence), anomaly detection (finding outlier/suspicious data -- potential poison), and vetting (reviewing suspicious data) -- cleaning the data (catching poison that gets in) -- complementing the provenance (though acknowledging hidden backdoors are hard to detect). The detection helped clean the data. A distinction case: the team distinguishes data poisoning (a training-time attack -- via the training data -- corrupting the model itself) from prompt injection (an inference-time attack -- via the input -- manipulating the model on that input) -- so they defend both (poisoning via provenance/vetting -- protecting the training data; prompt injection via input handling/guardrails -- protecting the inference) -- recognizing the different attack times and defenses. The distinction guided the defenses. The consolidated discipline the team documents: recognize data poisoning as a serious threat (corrupting the model via its training data -- degradation, bias, or hidden backdoors -- especially insidious backdoors that pass normal evaluation), defend primarily via data provenance (knowing and controlling the data sources -- using trusted, vetted sources -- reducing the poison's opportunity), complement with detection/filtering (deduplication, anomaly detection, vetting -- cleaning the data), protect all the stages (pretraining, fine-tuning -- fine-tuning especially, being smaller/easier to poison) and RAG knowledge bases (retrieval poisoning), distinguish it from prompt injection (training-time vs inference-time -- defend both), and monitor for poisoning effects -- because a model is shaped by its training data, so corrupting the data corrupts the model, making data poisoning (especially hidden backdoors) a serious, insidious supply-chain-style threat to model integrity, defended primarily by controlling and vetting the training data (provenance).