AFP engineering series · Chapter 19 of 25
Use models to organize fraud evidence without turning uncertain scores into automatic accusations or unsafe approvals.
A fluent explanation is not a calibrated risk estimate
An LLM can summarize a case and identify inconsistencies, but a confident narrative does not prove that a person committed fraud. The model may infer facts that are absent, misunderstand a document, or be influenced by instructions inside the case material. Keep evidence extraction, risk estimation, and enforcement as separate stages.
For Northstar, the assistant can assemble a review packet containing order records, submitted claims, and unresolved discrepancies. Every claim in the packet should point to a source record or be labeled as an inference. The final action policy should use verified facts and a defined review process rather than an unstructured model accusation.
Understand the base-rate arithmetic
Suppose a hypothetical workload contains 10,000 cases, with 1% actually fraudulent. A detector catches 90% of fraud and falsely flags 2% of legitimate cases. It finds 90 true positives and creates 198 false positives. Of 288 flagged cases, only 31.25% are fraud. These numbers are an illustrative calculation, not measured performance of any model.
This is why a seemingly low false-positive rate can overwhelm reviewers when fraud is rare. Precision depends on prevalence as well as sensitivity and false-positive rate. A balanced benchmark containing equal numbers of fraud and legitimate cases does not represent the review queue of a low-prevalence production system.
Calculate the operational workload
Illustrative python; adapt the persistence and identity adapters to your service.
def expected_counts(total, prevalence, tpr, fpr):
fraud = total * prevalence
legitimate = total - fraud
tp, fp = fraud * tpr, legitimate * fpr
return {"tp": tp, "fp": fp, "fn": fraud-tp,
"tn": legitimate-fp,
"precision": tp/(tp+fp) if tp+fp else None}
x = expected_counts(10000, .01, .90, .02)
assert x["tp"] == 90 and x["fp"] == 198
assert x["precision"] == .3125Translate flagged cases into review minutes using an observed handling-time distribution, not just an average. Complex cases often take longer, and queue delays can affect customers even when the final decision is correct. If the expected queue exceeds capacity, improve triage or staffing rather than silently auto-approving overflow.
Separate review from blocking. A flag may justify collecting more evidence without denying service immediately. The appropriate response depends on the action's reversibility and potential loss. Do not use one threshold for every workflow simply because it is easy to configure.
Prevent data leakage in evaluation
Split cases by time and by relevant entities so near-duplicate histories do not appear in both training and evaluation. Exclude information created after the decision point, such as a later chargeback outcome, unless the task explicitly predicts from that later time. Otherwise the model can appear accurate by using evidence unavailable in production.
Track label quality and delay. Some fraud outcomes take weeks to confirm, and some legitimate users appeal successfully. Treat labels as versioned evidence rather than eternal truth. Re-evaluate performance when the investigation process changes, because the observed label distribution may change even if user behavior does not.
Make the review packet inspectable
| Field | Purpose |
|---|---|
| Verified records | Facts from authoritative systems |
| Submitted claims | What the requester asserts |
| Discrepancies | Specific unresolved conflicts |
| Model inference | Clearly labeled interpretation |
| Recommended next evidence | What would reduce uncertainty |
| Decision history | Who acted and under which policy |
Require citations to actual record identifiers and reject invented references. Reviewers should be able to open the underlying evidence without trusting the summary. Keep sensitive raw documents in a restricted store and avoid copying them into every dashboard.
Design the interface to reduce automation bias. Show uncertainty and contradictory evidence, not just a large risk score. A reviewer needs the ability to disagree and record why. Analyze those disagreements to improve the system instead of treating them as errors by the human.
Validate thresholds and subgroup behavior
Measure precision, recall, false-positive burden, and loss-weighted outcomes at the operating threshold. Examine relevant languages and customer journeys to detect uneven performance. Do not infer sensitive traits from text to build a convenient segmentation scheme. Use appropriate, justified evaluation dimensions and protect the data used for analysis.
The threshold lab lets readers vary synthetic prevalence and detector rates. It intentionally shows expected counts rather than claiming measured model quality. For a real release, attach confidence intervals or uncertainty estimates and disclose dataset limitations. A small test set cannot support a precise claim about rare failures.
Worked review queue: an apparently strong detector overwhelms staff
Use the hypothetical 10,000-case workload with 1% fraud, 90% sensitivity, and 2% false-positive rate. The detector flags 288 cases. If the team can review 100 cases in the relevant period, 188 remain queued before considering variability in handling time. A dashboard showing 90% recall hides this operational bottleneck.
Do not solve the backlog by silently approving overflow or blocking every queued customer indefinitely. Separate cases by consequence and required evidence. Some can receive a low-risk restricted service while review continues; others must remain held. Allocate reviewers and improve evidence collection based on measured handling time, and communicate status to affected users.
Compare a stricter synthetic operating point with 70% sensitivity and 0.5% false-positive rate. It flags 70 true positives plus 49.5 expected false positives. The queue is smaller, but more fraud is missed. Expected fractional counts are normal in a planning model; observed case counts are integers. The correct choice depends on losses, reversibility, customer harm, and available controls beyond the detector.
Add a deterministic transaction limit that contains some missed fraud. This can change the acceptable detector tradeoff without changing the detector itself. Conversely, granting the assistant more powerful tools can make the same threshold unacceptable. Risk belongs to the composed workflow, not to the score in isolation.
Review a sample of both flagged and allowed cases to estimate missed outcomes and label quality. If you only investigate flagged cases, your evaluation can become biased toward what the detector already notices. Track appeals and delayed confirmations. This worked case shows why AFP needs calibrated measurement, sufficient review capacity, and hard action constraints rather than a single “fraud probability” displayed beside a model-generated explanation.
Keep enforcement proportionate and reversible where possible
Use the model to prioritize investigation and organize evidence, while hard transaction rules remain deterministic. Preserve appeal and correction paths. Record the reason for a restriction in language support staff can understand without exposing confidential detector internals.
Monitor confirmed fraud, customer harm from false blocks, review backlog, and time to resolution. Improving only recall can make the product unusable; improving only approval rate can increase losses. AFP succeeds when the decision process combines evidence, calibrated uncertainty, and controls matched to the consequence of being wrong.
Practice and continue
Open the interactive companion lab to inspect the assumptions and outcomes with synthetic data. Then return to the 25-chapter AFP learning path.
Sources and implementation scope
Primary references checked October 15, 2026. Framework details follow the linked documentation; the Northstar cases, policy contracts, and test designs are original illustrative designs. Python snippets labeled illustrative are not complete production integrations. ADK package metadata checked for this series reported 2.12.0; pin and test the version used by your application.