Why architecture matters here
Consider what multi-tenant infrastructure demands of a network. Tenant A and tenant B both use 10.0.0.0/24 — their packets must never meet. A VM in rack 3 live-migrates to rack 40 — its IP, its open TCP connections, and its L2 neighbors must survive the move. An orchestrator creates and destroys thousands of isolated networks per day — provisioning must be an API call, not a switch-configuration change ticket. Classic VLAN-based designs fail all three: the ID space is too small, L2 domains cannot span a routed fabric without spanning-tree fragility, and VLAN plumbing touches physical switch state.
The pre-VXLAN workaround — stretching VLANs with trunk links and running spanning tree across the datacenter — was operationally notorious: STP blocks redundant links (wasting half the fabric), converges slowly, and a single misbehaving switch can melt an entire broadcast domain. Meanwhile the routing world had already solved scale: BGP fabrics with ECMP use every link, converge fast, and fail gracefully. The strategic insight of overlay networking is to stop fighting this — build the physical network as pure L3, which routers are excellent at, and synthesize whatever L2 semantics tenants need in software at the edge.
That split is why VXLAN won so broadly: the underlay team runs a small, boring, uniform router fabric whose config barely changes, while the overlay is programmable — an SDN controller, a Kubernetes CNI, or EVPN configuration can create a tenant network in milliseconds without touching a spine. The cost is a new class of problems living between the layers — encapsulation overhead, visibility gaps, and control-plane state — which is exactly where the rest of this article spends its time.
The alternatives clarify the choice. NVGRE and STT lost to VXLAN largely on hardware support and the ECMP-friendliness of a UDP outer header. Geneve generalizes VXLAN with extensible options and is the default in OVN and NSX-T, but its architecture — VTEP-equivalent endpoints over a routed underlay — is the same lesson. Pure-L3 designs like Calico in BGP mode skip encapsulation entirely and are the right call when workloads only need routed connectivity and the fabric permits advertising pod routes. VXLAN occupies the middle: full L2 semantics when tenants need them, over any IP fabric, with an ecosystem of ASICs, NIC offloads, and EVPN implementations behind it.
The architecture: every piece explained
The encapsulation: an original Ethernet frame is wrapped in a VXLAN header (8 bytes, carrying the 24-bit VNI), inside a UDP datagram (destination port 4789), inside an outer IP packet addressed from the source VTEP to the destination VTEP, inside an outer Ethernet frame for the next hop. Total overhead: 50 bytes (54 with an outer VLAN tag). The outer UDP source port is not arbitrary — VTEPs compute it as a hash of the inner flow, which is what lets the underlay's ECMP spread different tenant flows across different equal-cost paths while keeping any single flow on one path, in order.
The VTEP owns the mapping between worlds. On transmit: look up the destination MAC within the frame's VNI; if the MAC-to-remote-VTEP binding is known, encapsulate and send unicast. On receive: validate the VNI, strip the headers, and deliver the inner frame to the local bridge. VTEPs live in software (Linux vxlan devices, OVS — the basis of most Kubernetes CNIs and OpenStack networking), in smartNICs that offload encap/decap per flow, or in switch ASICs for hardware VTEP designs where the top-of-rack encapsulates on behalf of bare-metal hosts.
The control plane answers 'which VTEP is behind MAC X in VNI Y'. The original answer — flood-and-learn over IP multicast, one group per VNI — works but drags multicast operations into the underlay and floods proportionally to segment size. The modern answer is EVPN: each VTEP runs BGP (usually via route reflectors) and advertises Type-2 routes (MAC/IP bindings, learned locally) and Type-3 routes (which VTEPs participate in which VNI, used to build ingress-replication lists for broadcast/unknown/multicast traffic). Learning becomes proactive: a VM's first ARP announcement turns into a BGP route, and every VTEP in the VNI knows the binding before traffic flows.
EVPN unlocks the refinements that make overlays production-grade: ARP suppression (the local VTEP answers ARP requests from its EVPN table, so most broadcasts never cross the fabric), MAC mobility (sequence-numbered route updates make live migration a clean route replacement, with dampening for flapping duplicates), and anycast gateways (every leaf hosts the same virtual gateway MAC/IP per subnet, so routed traffic exits at the nearest leaf instead of hair-pinning to a central gateway). Routing between VNIs and out of the fabric happens at leaf SVIs or border leaves — the deliberate ends of the L2 illusion.
End-to-end flow
Trace a first packet. VM A (VNI 5001, on host 1) pings VM B (same VNI, on host 40). A's stack needs B's MAC and broadcasts an ARP request. Host 1's VTEP intercepts it and checks its EVPN-fed neighbor table: B's MAC/IP binding is already present (B's VTEP advertised a Type-2 route when B came up), so the VTEP answers the ARP locally — nothing floods. If the binding were absent, the VTEP would replicate the ARP as unicast VXLAN packets to each VTEP in VNI 5001's ingress-replication list, built from Type-3 routes.
A now sends the ICMP frame to B's MAC. The VTEP looks up the MAC in VNI 5001, finds remote VTEP 40's loopback IP, and encapsulates: VXLAN header with VNI 5001, UDP with a source port hashed from the inner five-tuple, outer IP from VTEP 1's loopback to VTEP 40's loopback. The packet enters the leaf-spine underlay, which treats it as an ordinary UDP packet: the leaf hashes the outer header across four equal-cost spine uplinks — the hashed source port ensures different tenant flows take different spines — and two routed hops later it arrives at host 40. That VTEP validates VNI 5001, decapsulates, and delivers the pristine inner frame to B's virtual switch port. B replies; the reverse path works identically. Neither spine ever saw a tenant MAC.
Now the interesting event: B live-migrates to host 12. The moment B's virtual NIC attaches on host 12, that VTEP advertises a Type-2 route for B's MAC with a mobility sequence number one higher than host 40's advertisement. BGP propagates; every VTEP atomically updates B's next-hop from VTEP 40 to VTEP 12; host 40's VTEP withdraws its route. In-flight packets briefly land at host 40 and are dropped or re-forwarded depending on implementation, TCP retransmits cover the gap, and A's ping stream resumes within the control-plane convergence time — typically well under a second. B kept its IP, its MAC, and its connections; the underlay never changed at all.
The same flow underlies the platforms most engineers actually touch. Kubernetes CNIs like Flannel and Calico's VXLAN mode run a software VTEP per node, one VNI for the pod network, with the node routing table standing in for EVPN at small scale; OpenStack Neutron and NSX run the full multi-VNI, controller-fed version; and cloud provider VPCs are the same architecture with proprietary headers — an availability zone is a routed underlay, a VPC is an overlay segment, and a security group is policy at the virtual switch. Learn the VXLAN flow once and every overlay you meet afterward is a dialect.