AI in Disaster Response, in depth: grounded extraction, corroboration against rumours, crisis data protection and offline operation

Threat model: what makes a crisis different

Start with what is different from a normal production system.

The adversaries are varied. Scammers post fake donation drives and impersonate relief agencies. Pranksters file false rescue requests, which pull boats away from real ones. Political actors amplify rumours about looting, contaminated water or particular groups to inflame tension. And some real reports are simply wrong: a post about water rising on a street can be hours old by the time it is read. The pipeline must assume every single report could be false and still act fast on the ones that are true.

Reference architecture

The architecture separates three concerns. Situational awareness turns noisy inputs into a deduplicated, scored list of incidents for responders. Public information answers questions from the public using only official, current bulletins. Field resilience keeps both working when the network does not. The model never dispatches anything and never issues an alert; it prepares information for people who do.

Inputs carry their provenance through the whole pipeline. A report from a trusted field team, a transcribed hotline call, a public post and a river gauge reading are different kinds of evidence and must keep their source type, timestamp and channel attached. Interoperable alerting standards such as the OASIS Common Alerting Protocol (CAP) are useful for the output side, because responder systems and public alert channels already accept them.

Extraction that refuses to invent facts

The LLM's main job in situational awareness is extraction: turn a message into a structured report. The dangerous failure is fabrication. A model asked for a location will produce one, and a confident but wrong address sends a crew to the wrong street. The schema must make "unknown" a valid, encouraged answer, and validation must reject anything not grounded in the input text. The general risk is discussed in LLM hallucination risk; here is the disaster-specific defence.

REPORT_SCHEMA = {
  "need": ["rescue", "medical", "water", "food", "shelter", "info", "other"],
  "location_text": "str | null",   # verbatim span from the message, or null
  "people_count": "int | null",
  "urgency_cues": "list[str]",     # verbatim spans: 'water to roof', 'not breathing'
  "reported_at": "str | null",     # if the message states a time
}

SYSTEM = """Extract fields from the crisis message. Copy location and urgency
cues VERBATIM from the message. If a field is not stated, return null.
Never infer an address, landmark or count that is not written in the text."""

def grounded(span, message):
    return span is None or span.lower() in message.lower()

def extract(message, llm):
    out = llm.json(SYSTEM, message, schema=REPORT_SCHEMA, temperature=0)
    if not grounded(out["location_text"], message):
        out["location_text"] = None
        out["flags"] = out.get("flags", []) + ["location_not_in_source"]
    out["urgency_cues"] = [s for s in out["urgency_cues"] if grounded(s, message)]
    out["needs_followup"] = out["location_text"] is None and out["need"] in ("rescue", "medical")
    return out

The verbatim-span check is cheap and catches most location fabrication. Geocoding then happens in a separate, deterministic step against a gazetteer, which returns candidates with confidence rather than a single guess. Rescue and medical reports with no usable location go to a follow-up queue, where a human or an automated SMS asks for more detail. Translation follows the same rule: keep the original text beside the translation, so a bilingual reviewer can check any report that will drive a dispatch.

Corroboration and anti-poisoning

Corroboration is the main defence against both honest error and deliberate poisoning. Cluster reports that describe the same incident, then score each cluster by the independence and reliability of its sources, not by raw volume. Volume is exactly what a coordinated campaign can fake.

from math import prod
from statistics import mean

SOURCE_WEIGHT = {"field_team": 1.0, "sensor": 0.9, "hotline": 0.6, "sms": 0.5, "social": 0.25}

def cluster_score(reports, now):
    # Count independent sources: one account or phone number counts once.
    by_origin = {}
    for r in reports:
        w = SOURCE_WEIGHT[r["channel"]]
        if r["channel"] == "social" and r["account_age_days"] < 7:
            w *= 0.3                                  # fresh accounts weigh little
        age_h = (now - r["received_at"]).total_seconds() / 3600
        w *= 0.5 ** (age_h / 6)                        # half-life of six hours
        by_origin[r["origin_id"]] = max(by_origin.get(r["origin_id"], 0), w)
    score = 1 - prod(1 - w for w in by_origin.values())  # noisy-OR over origins
    return score

def campaign_signals(reports):
    texts = [r["text"] for r in reports if r["channel"] == "social"]
    near_dupes = share_near_duplicate(texts, threshold=0.9)   # MinHash or embeddings
    young = mean(r["account_age_days"] < 7 for r in reports if r["channel"] == "social")
    burst = max_posts_per_minute(reports)
    return {"near_dupe_share": near_dupes, "young_account_share": young, "burst": burst}

Noisy-OR over independent origins means one trusted field report outweighs a hundred copies of the same post from new accounts. The six-hour half-life is an example parameter for a fast-moving flood; tune it to the hazard. Campaign signals do not delete anything; they route the cluster to a manipulation-review queue, because a burst of near-identical posts can also be a real community copying a plea for help. Humans make that call. Low scores never suppress a life-safety report outright; they lower its priority and trigger a verification step, such as a call-back or a drone pass.

A grounded public assistant

Public-facing assistants answer questions such as where the nearest open shelter is, whether a road is passable, or how to purify water. Three rules keep them safe. First, retrieval is restricted to an allow-list of official, timestamped sources: emergency management bulletins, public health guidance and utility notices. Social media never grounds a public answer. Second, every answer states its source and time, and when the newest relevant bulletin is older than a threshold, the assistant says so rather than answering from stale data. Third, anything that sounds like an emergency gets the emergency number first and the assistant's help second; the assistant is not a dispatch channel and must say so.

Medical questions are bounded to published first-aid and public-health guidance, quoted rather than paraphrased where possible. The assistant must refuse to speculate about casualty numbers, causes or blame, which are the topics rumours cluster around. Test these behaviours before activation with a red-team set built from past events' rumours, in every language the assistant will serve.

Protecting data about affected people

Crisis data is unusually sensitive. A map of where people are trapped, injured or sheltering can also be used to target them: by looters, abusive partners, traffickers or, in conflict-affected settings, armed groups. Humanitarian practice treats this seriously; the ICRC's Handbook on Data Protection in Humanitarian Action is a standard reference, and it is worth reading before designing a pipeline.

In engineering terms: minimise at extraction, so names, phone numbers and health details are stored only where a responder needs them and are redacted elsewhere; aggregate before sharing, so partner dashboards show counts per area rather than individual points; keep access role-based and logged; set short retention tied to the operation's end; and never send raw crisis messages to a third-party model API without a data processing agreement that covers the use. Detection and redaction techniques are covered in PII handling for LLM systems. Treat aggregated location data about specific groups as sensitive in its own right, even when no individual is named.

Degraded and offline operation

Plan for the cloud to be unreachable from the places that need help most. A field edge node, a ruggedised laptop or small server with a quantised small language model, a speech-to-text model and a local copy of the gazetteer, can run extraction and translation offline and queue results for sync when a link returns. Accuracy will be lower than the cloud model's, so tag edge-produced reports with their origin and re-run them centrally when connectivity allows.

Centrally, build for the surge before it happens: reserved capacity or pre-agreed quota increases with model providers, a second provider configured and tested, and priority queues so rescue and medical messages are processed first when load is shed. The patterns for provider failover are in multi-region inference and provider outage recovery. Finally, keep a rule-based fallback, a keyword and regex classifier for rescue and medical terms, that runs even when every model is down. It is crude, but it degrades to something rather than nothing.

Regulatory position

Regulation follows the use. Under the EU AI Act, Annex III lists systems that evaluate and classify emergency calls, or dispatch or set priority in dispatching emergency first-response services and emergency healthcare patient triage, as high-risk. A pipeline that ranks rescue requests for dispatch is close to that line, and the human triage desk does not by itself move a system out of scope. If you operate in the EU, get the classification reviewed, and expect duties around risk management, logging, human oversight and accuracy; under the Omnibus timeline those Annex III duties apply from 2 December 2027. A public information assistant has lighter, transparency-type duties. Elsewhere, public-sector procurement rules and data protection law usually set the bar. Whatever the jurisdiction, the oversight design in human-in-the-loop controls is the practical core.

Worked example: a river flood

A regional emergency agency activates its pipeline during river flooding. In the first six hours it receives 41,000 social posts mentioning the event, 3,200 transcribed hotline calls and 260 field-team reports. Keyword prefiltering and extraction reduce the social stream to 5,800 posts containing a need, of which 1,900 state a location verbatim.

Clustering yields 740 candidate incidents. Corroboration scores place 85 above the dispatch-review threshold of 0.6; most of those include a hotline call or field report. One cluster of 1,100 near-identical posts claims a dam breach upstream. Its near-duplicate share is 0.93 and 70 percent of the accounts are under a week old, so it goes to manipulation review rather than the triage desk. The gauge readings and the dam operator's bulletin show no breach. The public assistant, grounded on official bulletins, answers "Is the dam breaking?" by quoting the operator's latest statement and its time, and the comms team issues a rebuttal.

Meanwhile 64 rescue requests had no usable location. Automated SMS follow-ups recover addresses for 38 of them within an hour; the remaining 26 go to human callers. Two field teams lose connectivity; their edge node keeps extracting voice reports and syncs 140 queued items when a cell site is restored. After the event, the agency deletes raw messages at the 30-day retention mark, keeps aggregate statistics, and runs a review of every dispatch the pipeline influenced.

Failure modes

• Fabricated locations. The model fills a missing address. Enforce verbatim spans and geocode deterministically.
• Volume equals truth. Ranking by post count rewards campaigns. Score independent origins with source weights.
• Stale grounding. The assistant cites a shelter that closed hours ago. Timestamp every source and refuse beyond a freshness limit.
• Silent suppression. A low-scored real rescue request is dropped. Lower priority and verify; never discard life-safety reports automatically.
• Data exposure. A shared map reveals where vulnerable people shelter. Aggregate before sharing and control access.
• Untested surge. Rate limits hit on day one. Load-test at expected peak and pre-arrange quota and a second provider.
• Language gaps. Extraction works well in the main language and poorly in minority ones, so those communities get slower help. Evaluate per language and route low-confidence items to bilingual reviewers.

Trade-offs

What to do next

1. Map your current information flow during an incident and mark where AI would prepare information and where humans decide.
2. Define the extraction schema with nullable fields and verbatim-span validation, and test it on messages from a past event.
3. Implement origin-based corroboration and campaign signals; route, never delete.
4. Build the public assistant on an allow-list of official, timestamped sources with a freshness cut-off, and red-team it with past rumours in every served language.
5. Write the data protection plan: minimisation, aggregation, access, retention and processor agreements.
6. Provision surge capacity and a tested second provider; package an edge node and a rule-based fallback.
7. Get the regulatory classification reviewed, then run a full exercise before the season your hazard peaks.