Why architecture matters here
Bad moderation is one of two things: too aggressive (users bounce because everything is refused) or too permissive (a violation ships and the team is on the news). The architecture matters because you must tune both sides simultaneously.
A layered pipeline lets you tune. Cheap heuristics catch the trivially bad without paying for an ML call. ML classifiers handle nuance. The policy engine encodes what your product actually cares about (a health app has different rules than a coding assistant). Humans handle ambiguity.
Without layers, your only lever is a single threshold, and moving it hurts one side to help the other. With layers, you tune each independently and prove the change with metrics.
The architecture: every piece explained
The top strip is the decision path. Input is a user prompt. Pre-classifier runs cheap heuristics — length, regex for obvious keywords, prior-bad-actor lookups. This catches 40-60% of clear violations at microsecond cost. ML classifier scores across categories — toxicity, PII, jailbreak, self-harm, weapons — with calibrated confidence. Policy engine combines the scores with product rules: what is acceptable in your product, what is required by jurisdiction, what user tier grants what latitude.
The middle row is the response path. Model call uses a system prompt reinforced by guardrails from the classifier output. Output classifier checks the response — sometimes the model produces something the input classifier missed. Escalation queue receives cases where the classifiers and policy engine cannot decide with high confidence; these route to Human review by trained moderators with rubrics and calibration checks.
The bottom rows are the learning loop. Feedback + labels capture moderator decisions as training data; hard examples are worth 10x easy ones. Metrics + audit track precision, recall, latency, and human-review SLA per category. Policy versioning + release train ensures every rule change is reviewed, tested against a golden set, staged, and only then promoted.
End-to-end flow
End-to-end: a user asks "how do I hack into my ex's Facebook?" The pre-classifier flags "hack into" as a soft keyword. The ML classifier returns high confidence on unauthorized-access intent. The policy engine returns block-with-explanation. The model is prompted to refuse politely with resources. The output classifier verifies the refusal is well-formed. No escalation needed. Now consider "how do I hack my morning routine?" — the pre-classifier flags the same keyword; the ML classifier sees benign intent (routine, morning); the policy engine allows; the model responds helpfully. Difference: nuance, not a keyword. Every decision is logged; sampling drives audit and moderator training the next week.