Why architecture matters here
The reason discovery matters is that in a cloud-native system, nothing has a stable address. Autoscaling adds and removes instances by the minute; a scheduler like Kubernetes kills a pod and starts a replacement on a different node with a different IP; a rolling deploy cycles the entire fleet through new addresses; a spot instance vanishes with two minutes' notice. Any scheme that assumes a service's location is knowable ahead of time is wrong within seconds of deployment. The location is emergent and constantly changing, so the system needs a runtime answer to 'where is X right now?' — and that answer must be fast, because every cross-service call depends on it, and correct, because routing to a dead instance is an outage.
This is why discovery is architectural rather than a library detail: it determines the failure and scaling behavior of every call in the system. A well-designed discovery layer lets you add capacity by simply starting instances that register themselves, roll out new versions by letting new instances register and old ones deregister, and survive instance failures because health checks route traffic away from the dead. A poorly designed one becomes a single point of failure whose outage takes down everything, or a source of subtle bugs where clients route to instances that no longer exist.
The second load-bearing concern is freshness versus stability. The registry is a distributed, eventually consistent view of a fast-moving fleet, and there is inherent tension between reacting quickly to change (so traffic doesn't hit dead instances) and not over-reacting to transient blips (so a momentary network hiccup doesn't evict a healthy instance and cause churn). Health-check intervals, TTLs, and deregistration timing are the knobs that set this balance, and getting them wrong produces either stale routing to dead nodes or flapping that destabilizes the whole mesh. The architecture has to make this trade-off explicit and tunable.
The third reason is the one teams discover during an incident: the discovery system's own availability is a load-bearing assumption for the entire platform. If every call requires a live lookup and the registry goes down, the naive design fails every call — discovery becomes the ultimate single point of failure. Serious designs therefore treat the registry as highly-available infrastructure and, crucially, make clients degrade gracefully: cache the last-known-good set of instances and keep using it when the registry is unreachable, favoring slightly stale routing over total failure. What discovery does when it is itself broken is the question that separates a resilient architecture from a fragile one.
The architecture: every piece explained
Top row: the registry and its inputs. A service instance, on startup, registers itself — posting its logical service name, address, port, and metadata (version, zone, capacity) to the registry, which is the source of truth mapping service names to the set of live instance addresses. Common implementations are Consul, etcd, ZooKeeper, or a platform-native one like Kubernetes' endpoints. The registry's job is to stay current, which it does with the help of health checks: the instance sends periodic heartbeats that renew a TTL, or the registry actively probes an endpoint, and an instance that stops passing checks is removed so clients stop being handed a dead address. A client that needs to call the service consults this system rather than any hardcoded address.
Middle row: the two discovery patterns and their abstractions. In client-side discovery, the client queries the registry directly, receives the list of healthy instances, and applies its own load-balancing (round-robin, least-connections, zone-aware) to pick one — giving the client full control at the cost of embedding discovery logic in every service. In server-side discovery, the client simply calls a stable endpoint — a load balancer, an API gateway, or a virtual IP — and that intermediary queries the registry and forwards to a healthy instance, keeping clients simple at the cost of an extra hop. Two widespread packagings sit on top: DNS-based discovery exposes the registry as DNS records so any client can resolve a name to current addresses, and a service-mesh sidecar (Envoy, Linkerd) intercepts the client's traffic and does discovery and load-balancing transparently, so application code calls a name and the sidecar handles the rest.
Bottom rows: keeping the view correct and cheap. Deregistration is the mirror of registration: on graceful shutdown an instance deregisters immediately so no traffic is sent to it during drain, and on ungraceful failure the health-check TTL expires and the registry evicts it. Caching and watches keep discovery fast and resilient: clients (or sidecars) cache the current instance set locally so every call isn't a registry round-trip, and subscribe to watches so the registry pushes updates when the set changes, rather than the client polling. The ops strip names the tuning that governs behavior: health-check interval and thresholds, the stale-entry TTL, the registry's own high-availability (it is quorum-replicated, not a single node), and — most important — the client's defined behavior when the registry is unreachable.
The choice between client-side and server-side discovery is the defining architectural decision, and it is really about where you want the routing intelligence to live. Client-side discovery gives each service sophisticated, application-aware load balancing and removes the extra network hop, but couples every service to the registry's client library and its update semantics — a polyglot fleet must implement discovery in every language. Server-side and mesh-based discovery centralize that logic in infrastructure the application never sees, so services stay thin and language-agnostic, at the price of an extra hop and a shared component to operate. The modern default gravitates toward the mesh sidecar precisely because it delivers client-side-quality routing without embedding it in application code — the best of both, paid for in operational complexity of the mesh itself.
End-to-end flow
Trace a request through a mesh-based deployment, the common modern shape. The checkout service needs to call payments. Every payments instance, on startup, registered with the registry — name payments, its pod IP and port, version v7, zone us-east-1a — and each renews a health TTL every few seconds. The registry thus holds the live, healthy set of payments instances at all times.
Discovery. Checkout makes a plain HTTP call to http://payments/charge. Its Envoy sidecar intercepts the call. The sidecar already holds a cached, watch-updated view of the payments instance set — pushed to it by the mesh control plane, which reads the registry — so there is no per-request registry round-trip. The sidecar selects a healthy instance using its configured policy: zone-aware routing prefers an instance in us-east-1a to save a cross-zone hop, and least-request load-balancing among those picks the least-busy one.
Call and health. The sidecar forwards to the chosen instance, adding mTLS, timeouts, and retry policy along the way. Suppose that instance had crashed a moment ago: its health TTL is about to expire, but the registry may not yet have evicted it. The sidecar's own active health checking and outlier detection catch the failing connection, eject that instance from its local pool, and retry the request against another healthy instance — so a single dead node produces a retried success, not a user-facing error. Meanwhile the registry's TTL expires and the dead instance is removed for everyone.
Scaling and deploy. Load rises and the autoscaler starts three new payments pods; each registers on startup, the control plane pushes the update to every sidecar's cache within a watch cycle, and traffic begins flowing to the new instances with no config change and no restart of callers. A rolling deploy of v8 works the same way: new-version instances register, old ones gracefully deregister as they drain, and the sidecars shift traffic across the transition. Now the stress test: the registry cluster has a partition and becomes briefly unreachable. Because the sidecars hold cached instance sets and are configured to serve stale-on-error, calls keep routing to the last-known-good instances rather than failing — the platform degrades to 'slightly stale routing' instead of 'total outage', which is exactly the failure-mode default a resilient discovery design must specify in advance.
Step back and notice what made every one of those transitions — a dead node, a scale-out, a rolling deploy, a registry partition — a non-event for the caller. In each case the application code did nothing: it kept calling http://payments/charge, and the discovery layer absorbed the change underneath it. That invisibility is the entire design goal. The instance that crashed was routed around by health checking and retry; the instances that were added registered themselves and appeared in the sidecars' caches through the control plane's watch push; the deploy shifted traffic by the same register/deregister mechanics; and the registry outage was survived because the caches held last-known-good state. A system without a discovery layer would have required a config change and a redeploy of callers for at least the scaling and version transitions, and would have simply failed the others. The lookup indirection — 'ask by name, get the current healthy set' — is what converts a continuously-moving fleet into something application code can treat as stable, and it is why discovery is infrastructure rather than an application concern.