Why it matters

Content moderation essential. Understanding shapes filter design.

Advertisement

The architecture

Classifier: text → harm categories.

Score per category.

Content scanner flowText inputprompt or responseClassifierharm categoriesActionblock or warnOpenAI Moderation API + Perspective API + Llama Guard commonly used
Scanner uses.
Advertisement

How it works end to end

Perspective API: Google. Toxicity detection.

OpenAI Moderation: free.

Custom: fine-tune on your categories.

Threshold tuning per false positive tolerance.