Fraud Prevention in LLM Applications: Verify Identity, Evidence, and Actions

Distinguish persuasive content from verified authority

An assistant handling invoices, account recovery, or refunds will encounter convincing claims. Fluency does not establish identity or entitlement. A message can sound like a senior executive, describe a plausible emergency, and include a realistic document while still being fraudulent. The application needs evidence from authenticated systems before it changes an account or initiates a payment.

The FBI's December 2024 advisory on generative AI and fraud describes synthetic text, images, audio, and video being used in deceptive schemes. It also recommends independently verifying a caller through a known contact channel. That is a historical advisory, not a current estimate of fraud prevalence. The workflow below translates the verification principle into an engineering design.

Make the assistant an evidence organizer

Consider a support agent receiving a request to refund an order to a new destination. Let the LLM extract the claimed order number, summarize the customer's explanation, and identify missing information. Mark each extracted value as unverified until the backend resolves it against the authenticated customer's records.

Store evidence with provenance: source system, record identifier, retrieval time, and relevant policy version. A screenshot of a payment is a submitted artifact; the payment processor's transaction record is a separate source. Do not collapse both into a single confident narrative. If sources conflict, show the conflict and route the case for investigation.

The model should not invent missing fields. An unknown amount, destination, or account owner must remain unknown. A structured response schema can represent these states explicitly, making it harder for downstream code to mistake an empty field or a plausible guess for confirmed data.

Separate eligibility, approval, and execution

OWASP's excessive agency guidance emphasizes limiting tool capabilities and permissions and enforcing authorization downstream. Apply that principle by making the assistant propose a refund while an independent service determines eligibility and executes the approved action.

The eligibility service should calculate refundable value from authoritative transactions, including previous refunds and pending reservations. Use integer minor units with explicit currency handling. Bind approval to a specific order, amount, currency, and destination, with an expiry and a unique operation identifier. If any field changes, require a new decision.

Use a transactional reservation or equivalent concurrency control before issuing the external request. An idempotency key helps with retries, but it does not by itself prevent two different requests from exceeding the same order's remaining refundable balance. Reconcile asynchronous processor results and handle uncertain outcomes without blindly issuing another payment.

Handle destination changes as a distinct event

A new bank destination or account recovery request should not inherit trust from a familiar writing style or an attached identity image. Verify the change through an established authenticated channel, using contact details already held by the service rather than details supplied in the suspicious request.

Keep the destination-change workflow separate from the payment workflow. An attacker should not be able to change a destination and immediately use the same conversation to authorize a transfer. Depending on the product, a hold period, additional verification, or independent review can provide time to detect a compromised account. Choose these controls from the actual risk model instead of copying arbitrary thresholds.

Use an LLM carefully in fraud analysis

An LLM can summarize a case for an analyst or explain which records disagree. Treat its fraud label as a fallible recommendation. The case material itself may contain misleading instructions, and a confident explanation can cite facts that are absent from the evidence. Require references to actual records and make the original evidence accessible to the reviewer.

Evaluate on time-separated cases to reduce leakage from later investigation results. Measure precision, recall, and false-positive impact at the operating threshold, with attention to the underlying fraud rate. A model can appear accurate by approving almost everything when fraud is rare. Track reviewer workload and successful appeals as operational outcomes, not just offline classification scores.

Test the failure paths before enabling money movement

Use synthetic accounts and a sandbox processor to test mismatched ownership, repeated submissions, parallel refund requests, expired approvals, changed destinations, unavailable verification services, and ambiguous processor timeouts. Confirm that a model-generated instruction cannot override a failed check. Re-run these cases whenever tool schemas or approval logic changes.

For incidents, provide a way to pause payment tools, revoke compromised credentials, and preserve a restricted audit trail linking proposal, evidence, approval, and processor result. Avoid logging full payment credentials in prompts or general analytics. Assign an operator to reconcile uncertain transactions and communicate through established support channels.

The goal is a workflow whose actions remain bounded even when a model is persuaded by a fraudulent story. Better detection helps prioritize cases; verified identities, independent authorization, and reliable transaction handling protect the final action.