Firestore architecture — collections, queries, realtime, security rules

The data model — documents, subcollections, and a key space that is flat underneath

Firestore stores documents, and documents live in collections. A document is a set of named fields with types -- string, number, boolean, timestamp, bytes, geopoint, null, a reference to another document, an array, or a nested map -- and it is capped at 1 MiB, counting field names, values and a little per-field overhead. There is no schema. Two documents in the same collection can have entirely different fields, and nothing in the database objects.

Paths alternate strictly: collection, document, collection, document. A document can own subcollections, and that is the only nesting the database understands beyond maps and arrays inside a single document. The important thing about a subcollection is what it is not: it is not stored inside its parent. Underneath, the key space is flat -- every document is keyed by its full path, and contiguous ranges of that key space are what get sharded across servers. Three consequences follow immediately, and all three surprise people.

First, deleting a document does not delete its subcollections. The parent vanishes; the children remain, still readable by full path, still counted in storage. Recursive deletion is something your code or the CLI does, one page at a time. Second, queries are shallow. A query over orders never descends into orders/{id}/lineItems, which turns subcollections into a deliberate cost lever: park the bulky, rarely-needed part of an entity in a subcollection and your list views stay cheap. Third, because the parent is only a path prefix, a document that does not exist can still have children -- so an intermediate path segment may be a phantom.

Collection group queries are the escape hatch: they query every collection sharing an ID at any depth, so lineItems across all orders is one query. They need their own index scope, which is the first hint of the rule that governs everything else here.

Native mode and Datastore mode — the choice you make once

One backend serves two APIs. Native mode is Firestore as most people mean it. Datastore mode serves the older Cloud Datastore API for applications with App Engine lineage, running them on the modern storage engine and quietly deleting the Megastore-era constraints those applications were written around -- eventually-consistent queries, entity-group write ceilings, cross-group transaction limits.

The difference that matters is not the query dialect, it is who is allowed to hold the connection. Native mode ships mobile and web SDKs that speak to the database directly from an untrusted client, and everything distinctive follows from that: real-time listeners, offline persistence, and security rules as a server-evaluated authorization layer. Datastore mode has none of those. Access is IAM at the API level, which means a trusted server principal, which means you are writing the backend Firestore Native was going to let you skip. Collection group queries are likewise a Native-mode feature.

Mode is fixed when the database is created. Multi-database support changes the practical shape of that constraint -- the escape hatch is standing up a second database in the project and migrating into it, not flipping a switch on the existing one. Where a conversion path exists at all it is a migration with preconditions and a maintenance window, so treat the choice as one you make once, at design time, with the client topology already decided.

The genuinely irreversible decision sits next to it: location. A database is created either in a single region or in a multi-region (nam5, eur3), and that cannot be changed afterwards for the life of the database. A multi-region commit needs acknowledgement from replicas in more than one region before it returns, which buys a higher availability commitment and costs write latency on every single write. Regional is cheaper and faster and fails with its region. Pick with the write path in mind, because you are picking for good.

Every query is an index walk

Firestore has no scan-and-filter execution mode. There is no query planner deciding between a seek and a table scan, because the table scan does not exist. Every query is served by walking a contiguous range of an index, and if no index can serve it the query is rejected outright with FAILED_PRECONDITION rather than being answered slowly.

The payoff is a latency model that is genuinely unusual: query cost is proportional to the size of the result set, not the size of the collection. Fetching twenty documents costs the same whether the collection holds ten thousand or ten billion. There is no gradual degradation as data grows, no point at which yesterday's query becomes today's incident. What you get instead is a hard wall at development time, which is a much better place to meet it.

Two kinds of index exist. Single-field indexes are created automatically for every field of every document -- ascending, descending, and, for array fields, an array-contains index. That is why a brand-new collection answers simple filters with no setup at all. Composite indexes cover everything else: any query that filters or orders on more than one field. You declare them in firestore.indexes.json and deploy them with the app, or you click the link the error message hands you and let the console write it for you. Field order and direction inside the index are part of its identity, as is its scope -- collection or collection group.

{
  "indexes": [
    {
      "collectionGroup": "orders",
      "queryScope": "COLLECTION",
      "fields": [
        { "fieldPath": "ownerUid",  "order": "ASCENDING" },
        { "fieldPath": "createdAt", "order": "DESCENDING" }
      ]
    }
  ]
}

Indexes are not free. Entries are storage you are billed for, and every write updates the document plus each index entry that document touches -- so a wide, heavily indexed document is a slower and more expensive write than a narrow one. Single-field index exemptions are the tool for that: turn indexing off for the large free-text blob or the big array you never filter on. There is also a ceiling on index entries per document, and an unbounded array in an indexed field is the reliable way to hit it, because each element produces its own entry. Treat indexes as schema: in source control, reviewed, deployed with the code.

What index-only execution forbids

Index-only execution is a bargain, and the constraints are the other half of it. They are worth knowing before you model the data, because most of them cannot be worked around later.

There are no joins. No subqueries, no lookups, nothing that reads a second collection on the server's behalf. You either denormalize the fields you need onto the document you are querying, or you make a second round trip from the client and accept both the latency and the reads. The N+1 that an ORM would hide from you is here an explicit design decision with a price attached.

Filters must collapse to one index range. Historically that meant range and inequality filters were restricted to a single field, which is the constraint most Firestore data models were shaped by. Inequalities across multiple fields are now supported, but they still require a matching composite index and they impose an ordering rule -- the inequality fields have to lead the sort. Disjunctive operators (in, not-in, array-contains-any, and or compositions) are capped at a modest number of values, tens rather than thousands, because the backend expands them into that many index scans and merges the results.

There is no full-text search. Prefix matching can be faked with a range scan between a prefix and the prefix followed by a high sentinel character, and that is the whole of it -- no tokenisation, no stemming, no relevance. Real search means mirroring documents into a search service through a write trigger. Vector search over embeddings does exist as a first-class nearest-neighbour query, with its own index type.

Aggregation is narrow but useful. count(), sum() and avg() execute on the server and are billed per batch of index entries scanned rather than per matching document, so counting a large result set costs a small fraction of reading it. There is still no GROUP BY; grouped rollups are something you maintain yourself on write, or export to BigQuery for.

Finally, ignore offset(). It is implemented by reading and discarding the skipped documents, and you are billed for every one of them. Page with cursors -- startAfter(lastSnapshot) -- which resumes the index walk at a position instead of counting to it.

Real-time listeners and what a watch stream actually costs

A real-time listener is a query you leave running. onSnapshot() registers the query on the server, which watches the same index range the one-shot query would have walked and pushes changes as they commit. All of a client's listeners are multiplexed over a single bidirectional stream, so twenty listeners is one connection, not twenty.

const q = query(
  collection(db, 'orders'),
  where('ownerUid', '==', uid),
  orderBy('createdAt', 'desc'),
  limit(20)
);
const unsub = onSnapshot(q, (snap) => {
  snap.docChanges().forEach((c) => apply(c.type, c.doc));  // added | modified | removed
});

Each callback carries the full current result set and a delta -- docChanges() reports what was added, modified or removed since the last callback, including reordering within the query's sort. That is the part worth appreciating: the diff a UI needs is computed for you, against a query, on the server side of a possibly flaky connection.

The cost model is where teams get hurt. The initial snapshot bills one document read for every document in the result. After that, you are billed one read per document delivered, so a listener on a document that changes a hundred times a minute costs a hundred reads a minute, per client listening. Nothing is billed while nothing changes -- an idle listener is free -- but the multiplier is the number of connected clients, and that is exactly the number that grows when a product succeeds. Reconnection resumes from a token and replays only what changed while you were away, though a listener that has been disconnected long enough (the documented threshold is half an hour) is re-billed as a fresh query.

Three habits keep this sane. Always attach limit() to a listened query, and never listen to an unbounded collection. Split volatile fields into their own small document so the stream carries only what actually moves, rather than re-delivering a fat document because one counter ticked. And for data that changes rarely, a periodic re-read is cheaper than a permanent watch. Server-side reactions belong somewhere else entirely: Cloud Functions triggers fire on document writes with at-least-once delivery, which means handlers must be idempotent.

Transactions, batches, and the retry semantics you have to code for

runTransaction() gives you an atomic read-modify-write: read some documents, compute, write, and either all of it lands or none of it does. One structural rule -- all reads must happen before any write inside the transaction, because the write set is only sent at commit.

The concurrency mechanism differs by SDK, and the difference is not cosmetic. Server SDKs are pessimistic: reads acquire locks, and a competing transaction waits, then fails if it waits too long. Mobile and web SDKs are optimistic: nothing is locked, the commit carries the versions of everything read, and the server rejects it if any of them moved. The SDK then re-runs your function from the top, a bounded number of times, before giving up.

That retry is the thing to design around. Your transaction body can execute more than once, so it must be a pure function of what it reads -- no HTTP calls, no logging that implies it happened, no mutation of application state outside the transaction, no random values you then persist as if they were decided once. Idempotence is not a nicety here; it is the contract.

Batched writes are the other atomic primitive. A WriteBatch applies many writes as a unit, but it never reads, so it has no version to conflict on and no retry semantics to reason about. It is bounded both by an operation count in the low hundreds and by total request size, which makes batch sizing a real design constraint for imports rather than a tuning knob. For bulk loads that do not need atomicity, BulkWriter parallelises writes and handles retry and backoff itself, and is almost always the better tool.

Contention is the failure mode you will actually meet. Transactions touching the same document serialise against each other, so a document under sustained concurrent update produces ABORTED errors and climbing latency. The canonical fix is the distributed counter: shard the value across N documents, increment a randomly chosen shard, and sum all N on read. N is a straight trade of write throughput against read cost. For a plain increment with no other logic, FieldValue.increment() avoids the read-modify-write round trip entirely -- though it does not lift the per-document write ceiling, only the transaction overhead.

Security rules are the authorization layer, and they are not filters

In Native mode the client holds the connection, so the authorization boundary has to live on the server side of it. Security rules are that boundary -- a declarative language evaluated by Firestore on every single request from a client SDK. They are not a client-side convenience and they are not advisory.

rules_version = '2';
service cloud.firestore {
  match /databases/{database}/documents {
    match /orders/{orderId} {
      allow read: if request.auth != null
                  && resource.data.ownerUid == request.auth.uid;
      allow create: if request.auth != null
                  && request.resource.data.ownerUid == request.auth.uid
                  && request.resource.data.total is number;
      allow update, delete: if false;
    }
  }
}

Rules see request.auth (the verified identity token, including custom claims), resource.data (the document as stored) and request.resource.data (the document as proposed). That last one makes rules do double duty: in a schemaless store, the write rule is the only place a type or a required field can actually be enforced.

Rules are not filters. This is the single most misunderstood thing about them. For a query, the server must be able to prove, from the query's own constraints, that every document it could return is permitted -- it will not evaluate the rule per document and quietly drop the rest. An ownership rule therefore rejects a whole-collection query outright, and the client must add the matching where('ownerUid', '==', uid) before the same rule will allow it. Access control and query shape are coupled by design.

Two more traps. Matches do not cascade: a rule on /orders/{orderId} says nothing about /orders/{orderId}/lineItems/{id} unless you write a recursive wildcard, and the default is deny, so this fails closed rather than open -- but the inverse mistake, a broad {document=**} wildcard at the root, fails wide. And get()/exists() lookups into other documents work, but they are billed as document reads and capped per request, so a rule chain that walks a membership document to find a role costs money and latency on every operation. Denormalise the role onto the document, or put it in a custom claim on the token.

Note what rules do not cover: Admin and server SDKs bypass them completely, authorised by IAM. Your Cloud Functions can do anything. Rules protect you from your users, not from your own backend.

Hotspots — random IDs, monotonic index fields, and the 500/50/5 ramp

Firestore shards a flat, ordered key space into contiguous ranges and moves those ranges across servers as load demands. The general consequence -- that keys clustered at one end of the range concentrate load on one server -- is the same mechanic covered in Spanner's architecture. What is worth spelling out here is the Firestore-specific surface, because there are two independent hotspots and people usually only defend against one.

The first is the document ID. The SDK's auto-generated IDs are random, and that is not a convenience -- it is a load spreader. Substituting a timestamp, a monotonic counter or a zero-padded sequence as the ID pushes every insert into the same key range, and no amount of backend splitting helps because the hot end keeps moving forward. If you need sortable IDs, put the sort key in a field, not in the key.

The second is the index, and it survives random document IDs entirely. An indexed field whose values increase monotonically -- createdAt is the universal example -- means every write appends to the same end of that index's key range, so the index is hot even though the documents are spread evenly. Google's documented guidance caps writes to a collection with sequential values in an indexed field at a few hundred per second for this reason. The fixes are to exempt the field from indexing if you never query on it, or to prefix the index key with a small random or hashed shard value and fan the query out across shards.

Related, and often mistaken for a quota: the guidance of one sustained write per second to a single document. It is a contention threshold, not a rate limiter. A burst of writes to one document is fine. Sustained concurrent writes are not, because each one has to update the document and all its index entries under the same lock, and past that rate you see retries and ABORTED rather than throttling. The distributed counter exists precisely to move past it.

Finally, ramping. The backend splits ranges in response to observed load, and splitting takes time, so the documented 500/50/5 rule applies to new collections and bulk loads: start at around 500 operations per second, then raise the ceiling by 50% every 5 minutes. A load test that opens at full throughput is measuring split lag, not the database.

Offline persistence and latency compensation

Offline support is not a caching layer bolted on top of the SDK; it is how the mobile SDKs work by default, with the network treated as an optimisation. On Android and iOS local persistence is on unless you turn it off. On the web it is opt-in, backed by IndexedDB, with a mode that coordinates across browser tabs.

Reads and queries run against the local cache. The same query semantics execute locally over the subset of documents this client has already seen -- which is the honest caveat: offline results are complete only with respect to the cache, so a query offline can legitimately return fewer documents than the same query online, and nothing signals that except the snapshot's fromCache metadata.

Writes go into a durable local mutation queue and are applied to the local view immediately. Your listeners fire straight away with metadata.hasPendingWrites set, which is latency compensation: the UI updates at the speed of local storage, then reconciles when the server acknowledges. The queue survives app restarts and replays in order on reconnect.

Conflict resolution is deliberately simple, and you should know which simple it is: last write wins at the server. There is no merge and no vector clock. A field set offline on Monday overwrites whatever the server did on Tuesday when the client finally reconnects. Commutative operations are the defence -- FieldValue.increment(), arrayUnion() and arrayRemove() are applied server-side against current state, so they survive replay in a way that a blind field assignment does not. Transactions, by contrast, need a round trip to read versions, so they do not queue offline; they fail.

The cache is bounded, with LRU eviction and a configurable size threshold, and cache hits are not billed -- though the read that put the document there was, and a listener re-established after a long disconnect is re-billed as a fresh query rather than resumed for free. Two operational implications follow. A client that has been offline for a week can reconnect with a large queue of writes, so server-side triggers must tolerate a burst of stale mutations. And security rules are evaluated at replay time, against the current state of the database -- so a rule that encodes a state machine, or checks request.time, will reject mutations that were perfectly valid when the user made them. That behaviour is correct, and it needs handling in the client rather than in the rules.

Billing by document read, not by byte

Firestore bills document reads, document writes, document deletes, stored bytes (documents plus index entries plus metadata) and network egress. The dimension that shapes every design decision is the first one, and specifically the unit: one read is one document delivered, regardless of its size.

That inverts habits carried from row-oriented databases. Ten thousand tiny documents cost ten thousand reads; one document holding the same data costs one. Denormalising into larger documents, or maintaining a summary document that a dashboard reads once instead of aggregating client-side, is not a micro-optimisation here -- it is often a change of an order of magnitude. Writes are counted per document too, so writing forty fields costs the same one write as writing one, which pushes the same direction.

The reads that surprise people are the ones that do not look like reads. A query matching nothing still bills a minimum of one read. offset() bills every skipped document. Each get() in a security rule is a read, on every request that evaluates it. Every document a listener delivers is a read, multiplied by connected clients. Deleting a collection bills a read for each document you enumerate and a delete for each one you remove. Cloud Function triggers that read neighbouring documents multiply the whole thing again.

Put together, the pathological pattern is easy to recognise once named: an unbounded query behind a list view, a query fired per keystroke, or a listener on a hot shared collection fanned out across every connected user. Firestore has no cost estimator and no query cost preview -- a bad query is discovered in the usage metrics or on the invoice.

The controls are unglamorous and effective. Put limit() on every query without exception, paginate with cursors, count with aggregation queries instead of fetching, use TTL policies to shed data that has aged out rather than accumulating index entries forever, exempt fields you never query from indexing, and wire budget alerts before launch rather than after. There is a free daily allowance of reads, writes and deletes that comfortably covers development, which is precisely why the first real bill is the one that shocks.

Firestore against Spanner, Bigtable, Cloud SQL and AlloyDB

Firestore is not the general-purpose database of Google Cloud, and most of its bad outcomes come from being used as one. It is worth being blunt about where the neighbours own the ground.

Spanner is the relational, SQL, externally-consistent option: joins, ad hoc queries, transactions spanning many rows across regions with TrueTime underneath. It costs more, it expects you to size compute, and it has no client SDK story -- but if the query shapes are not knowable in advance, that is the point at which Firestore's index-only model stops being a bargain. Bigtable is the other extreme: a wide-column store tuned for enormous key-ordered scans and single-key lookups, with no secondary indexes and atomicity only within a row. Cloud SQL and AlloyDB are real MySQL and PostgreSQL, and remain the right answer whenever the workload wants a query planner, joins, or the operational vocabulary a team already has.

Firestore earns its place when a specific set of things are true at once. Untrusted clients talk to the database directly, so security rules replace an API tier you would otherwise build and operate. Synchronisation and offline behaviour are product requirements rather than nice-to-haves. Access is predominantly by key or by a small number of query shapes known at design time. And the working set is documents that a user or a session owns, rather than the whole corpus.

It stops earning it when reporting shows up. Analytics, grouped rollups, ad hoc slicing and anything resembling a join belong in BigQuery, fed by a streaming export or scheduled snapshots -- a pattern common enough to be a supported product rather than a workaround. The same applies to per-byte economics: a workload that reads millions of small documents to compute one number is paying Firestore's pricing model to do a job its pricing model was designed to discourage. Managed export and import, scheduled backups and point-in-time recovery cover the operational side, and the export path is also how most teams get their data somewhere it can be queried properly.