Why architecture matters here
Detection quality is downstream of data quality, and the dependency is brutal. A detection for suspicious PowerShell is worthless if endpoint logs arrive with the command line truncated; an impossible-travel rule cannot fire if VPN logs land with unparsed source IPs; a correlation across authentication and process events dies if the two sources disagree about hostname formatting. In practice the majority of failed detections fail in the pipeline — collection gaps, parsing breakage, schema mismatches — not in the detection logic. Architecture matters because it determines whether those failures are visible (a parser-drift alarm at 9 a.m.) or silent (a missed lateral movement discovered by the ransom note).
The second force is economics at volume. Security telemetry grows faster than budgets — an EDR rollout or a cloud audit-log expansion can triple daily gigabytes overnight — and most SIEM pricing punishes ingestion. Architecture is the cost lever: filtering and routing at the collection tier (full-fidelity to cheap object storage, detection-relevant subset to the expensive hot tier), tiered retention aligned to real query patterns (triage searches last 30 days; incident response and compliance read cold storage), and schema-on-ingest normalization paid once instead of at every query. Teams that skip this design end up choosing between blindness (dropping sources) and budget crisis — usually alternating.
Third, the human bottleneck is the binding constraint. A SOC analyst can meaningfully triage a few dozen alerts per shift; the pipeline's job is to spend that scarce attention on maximum true-positive probability. Everything downstream of detection — dedupe, risk scoring, context enrichment, automated playbooks that close the trivially explainable — exists to protect the analyst's attention, and a SIEM that fires 4,000 alerts a day into a four-person queue has architected its own blindness regardless of how good its rules are.
The architecture: every piece explained
Top row: the data path. Sources span endpoint agents (EDR process, file, and network events), cloud control planes (CloudTrail, Azure activity, GCP audit), identity providers (login and MFA events — the highest-value stream per byte in the modern stack), SaaS audit logs, and network telemetry (DNS, flow, proxy). The collection tier is engineered plumbing, not an afterthought: agents and forwarders with local buffering and back-pressure so a SIEM outage does not drop endpoint history, syslog receivers behind load balancers, and API pullers with checkpoint cursors for SaaS sources. Two properties are non-negotiable — delivery accounting (every source has an expected-volume heartbeat, so silence pages someone) and at-least-once semantics with downstream dedupe.
Normalization parses each format into a common schema — ECS or OCSF — mapping every source's spelling of 'user logged in' to the same fields (user.name, source.ip, event.action). This is the highest-leverage stage in the pipeline: one schema means one detection for 'authentication from a new country' covers VPN, IdP, and cloud console simultaneously. Unparseable events are never dropped — they route to a quarantine index with a parser-failure-rate alarm, because parser drift after a vendor log-format change is among the most common silent-blindness causes. Enrichment joins context while the event is in flight: identity resolution (this username is that employee, in this department, whose manager is X), asset context (this host is a domain controller, not a kiosk), geo and ASN for IPs, and threat-intel indicator matches — precomputed at ingest so detections and analysts get context without N+1 lookups at 3 a.m.
Middle row: three detection surfaces with different physics. Stream detections evaluate sigma-style rules per event in real time — known-bad patterns (encoded PowerShell spawning from Office, a canary-token access) where single-event evidence suffices and latency should be seconds. The correlation engine holds windowed state to match sequences: N failed logins then a success from the same IP within 10 minutes; a new-country login followed by an MFA-method change followed by a mass mailbox-rule creation. Windowed state is memory, so window sizes and cardinality are engineering budgets, not free parameters. Batch analytics runs on the warm store — UEBA baselines (this service account's normal token volume, learned over 30 days), rarity scoring (first-ever admin action for this user), and scheduled hunts — trading hours of latency for statistical depth streams cannot afford. Tiered storage underlies all of it: hot indexed search for triage (7–30 days), warm columnar for analytics (90–180 days), cold object storage with on-demand rehydration for the year-plus compliance and IR tail.
Bottom row: from detections to decisions. The alert pipeline dedupes (one noisy host emits one rolled-up alert, not 400), correlates related findings into cases, scores risk from rule severity × asset criticality × identity privilege × intel confidence, and routes by score. SOAR playbooks automate the mechanical tier: auto-enrich every alert (recent host activity, user's other sessions), auto-close the explainable (impossible travel where the second 'country' is the corporate VPN egress), and execute containment with approval gates (isolate host, revoke sessions, disable account). Case management holds the human workflow — assignment, evidence, timeline, disposition — and its dispositions feed detection tuning, closing the loop.
End-to-end flow
Trace a phished credential through the stack. Step 1 — the signals exist. An employee enters credentials on a lookalike page at 09:14; the attacker logs in from a foreign ASN at 09:31, passes MFA via a fatigue-prompt approval, registers a new MFA device at 09:38, and begins mailbox-rule creation and SharePoint crawling by 09:45. Every step emitted telemetry: IdP login and MFA events, mailbox audit records, SharePoint access logs. Whether that becomes detection or archive is now purely a pipeline question.
Step 2 — ingest. The IdP puller delivers login events within ~90 seconds; normalization maps them to common auth fields; enrichment stamps geo (login country: new for this user), ASN reputation (hosting provider, not residential), identity context (finance department — high data access), and device posture (MFA approval from a device first seen today). Step 3 — detections converge. A stream rule fires low-severity on 'successful login from new country + hosting ASN' at 09:32. The correlation engine, holding a 20-minute window keyed on user, matches the sequence 'new-geo login → new MFA device registration' at 09:38 — a high-confidence account-takeover pattern. At 10:00 the UEBA batch pass adds that mailbox-rule creation volume is a 30-day anomaly for this user. The alert pipeline folds all three into one case, scored critical: privileged-data user, corroborated by three independent surfaces.
Step 4 — automated first response. The SOAR playbook for account-takeover executes: enrich the case with all sessions for the user (two active — one corporate device, one from the hosting ASN), auto-revoke the suspicious session's tokens, force reauthentication, and quarantine the new mailbox rules — reversible actions pre-approved for critical ATO cases; disabling the account outright waits for a human. Elapsed: attacker had 9 minutes of mailbox access, not a weekend. Step 5 — triage and closure. The analyst confirms from the case timeline (already assembled: logins, MFA change, rules, SharePoint access paths), disables the account, resets credentials and MFA, and sweeps cold storage for the sender's phishing pattern across other mailboxes — a rehydration query over 90 days. Disposition: true positive; the correlation rule's confidence weighting gets reinforced, and one noisy geo-rule exception (the user's legitimate travel next month) is encoded rather than muted. The pipeline turned four log streams into one nine-minute containment.