Actor Model Architecture in Depth

The actor model replaces shared memory with private state and messages. An actor owns some state, has an address, and reacts to one message at a time: it may change its own state, send messages to addresses it knows, and create new actors. Nothing else can touch its state, so the races, torn reads and lock ordering bugs of shared-memory threading cannot happen inside an actor.

This article is about the architecture you build with it. Once a system has thousands or millions of actors spread over several machines, the hard questions are no longer about one actor. They are about how big an actor should be, who restarts it when it fails, which node it lives on, how its state survives a crash or a rebalance, what happens to messages in flight, and how you notice when one actor becomes a bottleneck for everyone else. By the end you should be able to sketch an actor topology for a real workload, size it, and name the failure modes you need to test before production.

Actor architecture: supervision and dispatch inside each node, sharding, membership and an event journal across the cluster.

The components, stated precisely

Start with the moving parts, because several common descriptions of them are slightly wrong.

Actor. Private state plus a behaviour, a function from (state, message) to (new state, effects). An actor processes messages sequentially, so its handler never runs concurrently with itself. Different actors run in parallel.

Mailbox. The queue of messages waiting for an actor. In Erlang and in Akka's default configuration the mailbox is unbounded: a slow consumer simply accumulates memory until something breaks. Bounded mailboxes exist, but they are opt-in, which is why backpressure is a design decision and not a property you get for free. Ordering is also weaker than 'FIFO' suggests. The guarantee in both Erlang and Akka is per pair: messages sent from one actor to another arrive in the order they were sent. Messages from two different senders can interleave in any order, and Erlang's selective receive can deliberately process a later message before an earlier one.

Dispatcher or scheduler. A small pool of threads that runs whichever actors have mail. An actor holds a thread only while handling one message, which is how a node with 16 cores can host a million actors. The flip side: an actor that blocks on I/O or a lock holds a pool thread hostage. Akka lets you put blocking work on a separate dispatcher; Erlang's schedulers preempt processes after a budget of reductions, so one busy process cannot starve the rest.

Supervisor. The parent of an actor, which decides what happens when the child fails. Sharding and membership decide where actors live in a cluster, and the journal holds durable state for actors that must survive moves.

The components, stated precisely

Start with the moving parts, because several common descriptions of them are slightly wrong.

Granularity: what one actor should own

The first architectural decision is granularity: what does one actor represent? The rule that works is one actor per independent unit of consistency. If two pieces of state must change together atomically, they belong in the same actor, because an actor's message handler is the only transaction boundary the model gives you. If they can change independently, separate actors let them proceed in parallel.

Good entity actors map to nouns in the domain: a shopping cart, a bank account, a device, a chat room, a game match. Each sees a modest message rate and holds a bounded amount of state. Bad actor designs come in two shapes. The god actor owns too much, for example one actor for 'all accounts', and becomes a serial bottleneck because every message waits in one mailbox. The confetti design splits state that must stay consistent across many actors, then rebuilds transactions on top with sagas and compensation messages, which is distributed-systems work you chose to do inside one process.

A quick test: take your busiest entity and estimate its peak message rate. A single actor can typically handle tens of thousands to a few hundred thousand simple messages per second, depending on the work per message. If one entity, such as a hot product's inventory counter during a sale, needs more than that, you must split it (for example into stock buckets that each own part of the inventory) or move that hot path out of the actor model.

Supervision trees as failure design

Supervision is how actor systems turn failure into a design artefact. Instead of wrapping every call in defensive error handling, a child is allowed to crash, and its supervisor applies a policy: restart it with fresh state, resume it, stop it, or escalate the failure to its own parent. Restarting discards the possibly corrupt in-memory state, which is why 'let it crash' is safe only when the important state is either reconstructible or durable.

The tree is the architecture. Put actors that should fail together under the same supervisor, and give independent ones their own. Erlang/OTP names the strategies explicitly: one_for_one restarts only the failed child, one_for_all restarts every sibling, and rest_for_one restarts the failed child and those started after it, which suits a pipeline where later stages depend on earlier ones. Every supervisor also has a restart intensity: more than N restarts within T seconds and it gives up and fails itself, so a crash loop climbs the tree instead of spinning forever.

%% Erlang/OTP: a supervisor that restarts one session at a time and
%% gives up (escalates) after 5 restarts in 10 seconds.
init([]) ->
    SupFlags = #{strategy => one_for_one, intensity => 5, period => 10},
    Children = [#{id => session_registry,
                  start => {session_registry, start_link, []},
                  restart => permanent,
                  type => worker}],
    {ok, {SupFlags, Children}}.

The same idea in Akka Typed (Java), with exponential backoff so a child whose database is down does not hammer it with restarts:

Behavior<Command> supervised =
    Behaviors.supervise(DeviceActor.create(deviceId))
        .onFailure(Exception.class,
            SupervisorStrategy.restartWithBackoff(
                Duration.ofMillis(200),   // first delay
                Duration.ofSeconds(10),   // cap
                0.2));                    // jitter factor

Scaling out: sharding and virtual actors

On one node, an actor reference is a pointer to a mailbox. In a cluster, a reference may point to another machine, and the same tell works either way. This location transparency is useful, but the architecture still has to decide placement, and it has to guarantee that an entity exists exactly once. Two live copies of the same account actor, each accepting withdrawals, is the actor version of a split-brain database.

Cluster sharding is the standard answer. Entity IDs hash to a fixed number of shards; a coordinator (itself a cluster singleton in Akka) assigns each shard to one node; senders talk to a local shard region, which routes the message to wherever the entity lives and starts it on demand. When nodes join or leave, the coordinator moves shards, and the entities in a moved shard stop on the old node and restart on the new one, which is why sharded entities usually persist their state. Akka's guidance is roughly ten times as many shards as the maximum number of nodes, so rebalancing has fine-grained units to move.

Virtual actors, the model Microsoft Orleans introduced and used for Halo backend services, take this further: grains always conceptually exist, the runtime activates one on first message and deactivates it when idle, and the developer never creates or stops them. That removes lifecycle code but hides placement, so you have less control over co-location and memory.

// Akka Cluster Sharding (Java): one DeviceActor per device id, anywhere in the cluster.
EntityTypeKey<DeviceActor.Command> DEVICE =
    EntityTypeKey.create(DeviceActor.Command.class, "Device");

ClusterSharding sharding = ClusterSharding.get(system);
sharding.init(Entity.of(DEVICE, ctx -> DeviceActor.create(ctx.getEntityId())));

// Callers neither know nor care which node hosts the entity.
EntityRef<DeviceActor.Command> ref = sharding.entityRefFor(DEVICE, "dev-42");
ref.tell(new DeviceActor.Reading(21.7, Instant.now()));

Durable actors: event sourcing and the journal

An actor that must survive restarts, rebalances and deploys needs durable state. The common pattern is event sourcing: the actor validates a command, persists one or more events to a journal, and only then applies them to its in-memory state and replies. On restart it replays its events, usually starting from the latest snapshot so recovery does not replay a year of history. Akka Persistence (EventSourcedBehavior) implements this on top of a pluggable journal such as Cassandra or a relational database.

Two consequences shape the design. First, persistence must be single-writer: the journal relies on only one live instance of each persistence ID appending events, which is exactly what sharding plus correct downing guarantee. Second, the journal now sets your throughput. Persisting every telemetry reading from a million devices is a firehose; persisting only meaningful state transitions, and keeping raw readings in a stream or time-series store, keeps the journal sane.

Delivery guarantees and idempotency

Plain actor messaging is at-most-once. A tell to a remote actor can be lost if a node dies or a connection drops, and the sender gets no error. Systems that need stronger guarantees build them explicitly: the sender keeps the message until acknowledged and retries, and the receiver deduplicates by a message ID, which together give effectively-once processing. Akka ships this as its reliable delivery module, with producer and consumer controllers that track sequence numbers; in Erlang you write the acknowledgement protocol yourself.

Request-response needs timeouts on every ask, because the replying actor may have been moved or restarted between request and reply. And any message that crosses a process or service boundary should be idempotent, since retries are the only cure for loss.

Worked example: two million device actors

Take an IoT platform with 2 million devices, each reporting every 10 seconds, plus occasional commands from operators. One sharded DeviceActor per device holds the last reading, alert thresholds and connection status.

Message rate: 2,000,000 / 10 s = 200,000 readings per second. Across 10 nodes that is 20,000 per node per second, comfortably within a dispatcher's capacity if handlers do no blocking I/O.
Memory: assume about 2 KB of state per device plus actor overhead, call it 2.5 KB. 2,000,000 x 2.5 KB = 5 GB of live state, or 500 MB per node. Measure your real per-actor footprint; it is dominated by your state, not the runtime.
Shards: planning for up to 20 nodes gives 200 shards, about 10,000 devices each.
Persistence: persisting every reading would be 200,000 journal writes per second. Instead, persist events only when an alert state or configuration changes, perhaps a few hundred per second, and send raw readings to a stream for analytics.
Passivation: devices that have been silent for an hour are stopped to free memory; the next reading restarts them from their snapshot.

The sizing says the bottleneck is not CPU but the journal and the rebalancing window: when a node dies, about 200,000 devices must recover on other nodes, and recovery speed is set by snapshot reads.

Failure modes

Failure	Symptom	Mitigation
Mailbox growth	Memory climbs, latency rises, then the node dies	Bound mailboxes or use pull-based work, alert on mailbox depth
Blocking in a handler	Throughput collapses across unrelated actors	Separate dispatcher for blocking calls, or make I/O asynchronous
Hot entity	One actor's mailbox grows while others idle	Split the entity, or move that path out of actors
Split brain	Two copies of an entity accept writes	A downing strategy (Akka's split brain resolver, quorum or lease)
Restart storm	Children crash, restart, crash again	Backoff supervision and restart intensity limits
Lost message	Silent missing update after a node crash	Acknowledge, retry and deduplicate by ID
Slow recovery	Entities unavailable for minutes after a rebalance	Frequent snapshots, smaller shards, fewer events per entity

Operating an actor system

Actor systems hide queues everywhere, so operate them by watching queues. The four signals that matter are mailbox depth (per actor class, with the top offenders), message processing time, restart rate per supervisor, and dispatcher thread utilisation. Add cluster signals: membership changes, unreachable nodes, shard rebalances and journal write latency.

Deploys deserve special care. A rolling restart moves every shard at least once, so configure graceful shutdown to hand shards off, keep serialisation formats backward compatible for messages and persisted events, and never rename an event class without a migration adapter.

Trade-offs: when actors fit and when they do not

Actors fit workloads with many independent stateful entities and messages between them: sessions, devices, games, workflows, trading positions. They fit less well when the natural shape is a dataflow pipeline (streams with backpressure are clearer), when the work is pure parallel computation over arrays (use a thread pool or data-parallel library), or when the state really lives in a database and the service is stateless request handling, where actors add hops without adding safety.

Read on: the semantics of the actor model, actor runtime internals, Akka Typed interaction patterns, reactive streams backpressure and channels.

What to do next

List your domain's entities and mark which state must change atomically; that defines your actors.
Estimate the peak message rate of the busiest single entity and confirm one actor can handle it.
Draw the supervision tree and choose a strategy, restart intensity and backoff for each supervisor.
Decide which actors need durability, and persist state transitions rather than raw inputs.
Choose a shard count of about ten times your maximum node count and configure passivation.
Configure a downing strategy before your first multi-node deploy and test a network partition.
Add acknowledgement, retry and deduplication to any message whose loss would matter.
Export mailbox depth, processing time, restart rate and rebalance events, and alert on them.

Key takeaway: Actors remove shared-memory races, but the architecture lives in the decisions around them: one actor per unit of consistency, supervision trees that contain failure, sharding that gives each entity exactly one home, a journal for state that must survive moves, and explicit acknowledgement where messages must not be lost. Watch mailboxes, because every actor is a queue.