Deepfakes, in depth: how voice and video impersonation fraud works, why detection alone fails, and the process, identity and provenance controls that stop it

Anatomy of an impersonation attack

From the defender's side, a deepfake attack has three parts, and you can break any of them.

• Source material. Voice cloning needs recordings of the target. Executives provide plenty in earnings calls, conference talks, podcasts and social video. Face models need images or video. You cannot remove this material, so assume any public figure in your company can be imitated.
• The pretext. The fake person asks for something unusual: an urgent payment to a new account, a secret acquisition, an MFA reset because they are travelling. Almost every reported case pairs urgency with secrecy, because secrecy stops the victim checking with anyone else.
• The channel. Live calls, pre-recorded voice notes, video meetings, chat followed by a short call, or a camera feed during remote identity checks. Real-time video is harder to fake well than audio, which is why many attacks use audio, short clips, poor connections or cameras that are switched off.

The key point is that the fake media is only the delivery. The fraud is the request. If your process does not let a voice or face authorise a payment or a reset, the quality of the fake stops mattering.

Who gets targeted, and what removes the risk

Why detection alone fails

Deepfake detectors are classifiers trained on known generators. They have three structural problems.

• Generalisation. A detector learns the artifacts of the generators in its training set. A new generator, or an old one tuned for a while, produces different artifacts. Accuracy on public benchmarks routinely drops when the test set comes from generators the model never saw.
• The channel destroys evidence. Phone audio is narrow-band and compressed. Meeting software re-encodes video, lowers resolution and smooths faces. Many of the cues a detector needs are gone before it sees the media.
• Base rates. Even a good detector is mostly wrong when fakes are rare. The worked example below shows why.

Suppose one call in 10,000 to your finance team is a deepfake. Your detector catches 99 percent of fakes and wrongly flags 1 percent of real calls. Across 1,000,000 calls there are 100 fakes, and the detector catches 99 of them. It also flags 1 percent of the 999,900 real calls, which is 9,999 false alarms. Of the 10,098 flagged calls, under 1 percent are fakes. Staff learn to ignore the alert within a week. Detection has a place: it can rank media for human review, support a moderation team, or add evidence to an investigation. It should not be the control that decides whether money moves.

Controls that ignore how good the fake is

Process controls work because they do not ask anyone to judge whether a voice is real. They ask whether the request came through a channel the attacker cannot control. Encode them so a person under pressure does not have to remember them. A payment system can work out the required checks for each request and refuse to release funds until each check is recorded:

from dataclasses import dataclass

@dataclass
class PaymentRequest:
    amount: float
    currency: str
    beneficiary_id: str
    beneficiary_age_days: int      # how long this account has been on file
    channel: str                   # "erp", "email", "video_call", "phone", "chat"
    requester: str
    urgent: bool
    secrecy_requested: bool        # "keep this confidential", "do not tell X"

HIGH_RISK_CHANNELS = {"email", "video_call", "phone", "chat"}

def required_checks(req: PaymentRequest, single_limit=10_000):
    checks = []
    # The channel the request arrived on can be faked end to end,
    # so it never counts as verification.
    if req.channel in HIGH_RISK_CHANNELS:
        checks.append("callback to requester on a number from the HR directory, not from the request")
    if req.beneficiary_age_days < 30:
        checks.append("verify new beneficiary bank details by callback to a number already on file")
        checks.append("48h hold before first payment to this beneficiary")
    if req.amount >= single_limit:
        checks.append("second approver who was not on the original call or thread")
    if req.urgent and req.secrecy_requested:
        checks.append("escalate to fraud desk: urgency plus secrecy is the classic pretext")
    return checks

req = PaymentRequest(1_900_000, "HKD", "acct-7731", 0, "video_call",
                     "cfo@example.com", urgent=True, secrecy_requested=True)
for c in required_checks(req):
    print("-", c)

Run against a request shaped like the Arup case, it returns all five checks. The callback is the important one. Its number comes from your own directory, so an attacker who controls the meeting, the email thread and the caller ID still cannot answer it. The second approver must not have been on the original call, because a group call is exactly what the attacker staged. The new-payee hold matters because fraudulent payments usually go to accounts the company has never paid before, and a day or two gives the bank time to recall the money.

Help desks need the same treatment. Do not accept voice or video as proof for a password or MFA reset. Recover access through a device or security key the user already enrolled, or through their manager with an approval in the ticketing system. Phishing-resistant authenticators such as FIDO2 security keys and passkeys tie a login to a key the attacker does not hold, however good the voice.

Liveness and injection attacks in remote identity checks

Remote identity checks compare a selfie video with an ID document. They face two kinds of attack. A presentation attack holds something up to the real camera, such as a printed photo, a screen or a mask. Presentation attack detection (PAD) is tested under ISO/IEC 30107-3. An injection attack skips the camera. It feeds synthetic video into the app through a virtual camera driver, an emulator or a hooked capture API. PAD cannot help, because nothing was ever in front of a lens.

Injection defence works on the capture pipeline, not the face. Check device integrity and platform attestation where the OS offers it, detect virtual camera drivers and emulators, and use challenges the server randomises at capture time, such as a sequence of head turns, so a pre-rendered clip cannot match them. Ask vendors for independent test results on injection, not only on presentation attacks. Then send high-value or failed checks to a human reviewer, and include injected video in your own red team exercises against the onboarding flow, so you learn what the vendor misses before an attacker does.

Provenance for the media you publish

Provenance tries to prove where media came from, rather than spotting fakes. The C2PA standard, shown to users as Content Credentials, attaches a signed manifest to a file. A hard binding is a cryptographic hash of the exact bytes, so any edit breaks it. A soft binding is a watermark or content fingerprint that survives re-encoding and can be used to look up the manifest in a repository after the metadata has been stripped.

Use provenance for what you publish: sign official executive videos, press images and announcements, so you can show what is real when a fake appears. Its limits are equally important. Most media has no manifest, so missing credentials prove nothing. Uploads and screenshots strip metadata. Watermarks from AI generators are present only when the generator chose to add them, and open-weight models need not. The broader architecture is covered in LLM output provenance.

What the EU AI Act requires

Under the EU AI Act, Article 50 transparency duties have applied since 2 August 2026. Deployers who publish deepfakes must disclose that the content is artificially generated or manipulated, with lighter rules for clearly artistic or satirical work. Providers of generative systems must mark outputs in a machine-readable way. For systems placed on the market before 2 August 2026, the marking duty has a transition period that ends on 2 December 2026. The Digital Omnibus package deferred some obligations, notably high-risk deadlines, but most of Article 50 applies on its original date. Check the current text with counsel before relying on any date. See the EU AI Act for LLM teams for the wider obligations.

Operating the controls

Controls only work if people have practised them.

• Drills. Run simulated payment and reset requests against finance and help desk staff, with leadership approval and with any voice clone made only from executives who consented. Measure whether the callback happened, not whether anyone spotted the fake.
• A safe word is not enough. Shared phrases leak and are forgotten. Use them, if at all, on top of a callback, never in place of one.
• Fast recall. When a fraudulent payment is found, call the bank's fraud line within minutes and ask for a recall. Keep the number and the steps in the incident runbook.
• Meeting hygiene. Configure meeting and chat platforms to label participants from outside your tenant clearly, and teach staff that an internal-looking display name on an external account is a red flag worth a callback on its own.
• Keep evidence. Save meeting recordings, call logs, chat exports and headers. Police and insurers will ask for them.

Failure modes

The usual failures:

• Buying a detector and treating its silence as approval.
• Callback numbers taken from the email signature or meeting invite, which the attacker wrote.
• Exceptions for senior executives, who are the people most often imitated.
• Second approvers who were on the same staged call.
• Help desks that reset MFA after answering knowledge questions found on social media.
• KYC vendors tested only against presentation attacks.

Trade-offs

What to do next

1. List every action a voice, video or chat request can trigger today: payments, payee changes, resets, access.
2. Put a directory-sourced callback and a second approver in front of each one, and encode them as in the code above.
3. Add a hold before first payment to any new beneficiary.
4. Remove voice-only identity checks from the help desk and move executives to FIDO2 keys or passkeys.
5. Ask your KYC vendor for independent injection-attack test results.
6. Sign official executive media with C2PA and publish where people can check it.
7. Run a consented drill each quarter and track the callback rate.