Why architecture matters here
Tensor parallelism matters because it is the only parallelism strategy that directly attacks both constraints that pin a model to more than one GPU: capacity and single-token latency. Capacity is obvious — a model plus its KV cache that exceeds one device's memory must be sharded, and TP shards the weights themselves so each GPU holds a fraction. Latency is subtler: for interactive decoding, you generate one token at a time, and each token's forward pass is memory-bandwidth-bound; splitting each layer's weights across N GPUs means N times the aggregate memory bandwidth is applied to each token, so per-token latency drops roughly proportionally until communication overhead catches up. Pipeline parallelism, by contrast, helps throughput but adds pipeline-fill latency, and replication doesn't help a single request at all. For an SLM that must answer fast, TP is the tool.
The defining architectural fact — the one that governs every trade-off — is that TP's communication is synchronous and on the critical path of every token. The two all-reduces per layer are not background chatter; the next operation literally cannot start until the reduction completes, because it needs the full summed result. This has a hard consequence: TP scales well only across GPUs connected by very high bandwidth, low latency links — NVLink or NVSwitch within a node — and degrades sharply the moment the collective has to cross a slower boundary (PCIe, or worse, the network between nodes). That single property is why TP degree is almost always capped at the number of GPUs in one node (8, or 4), and why crossing nodes for TP is a well-known performance cliff rather than a free scaling knob.
Understanding this changes how you reason about deployment. The naive view — 'more GPUs, higher TP, faster' — is wrong past the fabric boundary: adding a ninth GPU across a slow link can make decoding slower because the all-reduce now dominates. The correct mental model is a balance between compute-split savings (linear in TP degree) and communication cost (grows with TP degree and explodes across slow links). Teams that internalize this pick the smallest TP degree that fits the model with acceptable latency, keep the group inside one high-bandwidth domain, and reach for pipeline or data parallelism to scale beyond it — rather than cranking TP until the collectives eat the gains.
For SLMs specifically, this balance has a pleasant sweet spot. Because small models are, well, small, they often fit in one or two GPUs' memory, so TP is used less to fit the model and more to cut latency — which means low TP degrees (2 or 4) inside a single node usually capture most of the win before communication overhead bites, and you rarely need the aggressive, node-spanning configurations that large-model serving forces. That makes TP one of the highest-leverage, lowest-risk knobs in the SLM serving toolbox: a modest degree on well-connected GPUs buys a real latency reduction with a communication tax you can keep small by construction.
The architecture: every piece explained
Top row: how attention shards. The QKV projection is column-parallel — the weight matrix is split by output columns, so each rank computes the Q, K, and V for a subset of the attention heads. This is the natural cut point because attention is already head-parallel: each head attends independently, so giving rank 0 heads 0–7 and rank 1 heads 8–15 requires no communication during the attention computation itself — each rank does its heads' scores and weighted sums entirely locally. The attention output projection that recombines heads is row-parallel: each rank holds the rows corresponding to its heads and computes a partial contribution to the full hidden vector, and an all-reduce sums those partials so every rank ends with the complete attention output. One communication, placed exactly where the heads must recombine.
Middle-into-top row: the MLP mirrors this. The up/gate projection (the expansion to the larger intermediate dimension, with the gating for SwiGLU-style MLPs) is column-parallel — each rank computes a slice of the intermediate activations, applies the nonlinearity locally (activations are element-wise, so no sync), and the down projection back to the hidden dimension is row-parallel, ending in the second all-reduce. So the pattern per transformer layer is: column-parallel → local work → row-parallel → all-reduce, twice (once for attention, once for MLP). Two all-reduces per layer is the entire synchronous communication budget, and it is fixed by the architecture, not by the input.
Middle row: the plumbing. NCCL collectives implement all-reduce (and all-gather, used by some variants) with bandwidth-optimal ring or tree algorithms across the ranks. NVLink/NVSwitch or the node fabric is the physical substrate whose bandwidth and latency determine whether those collectives are cheap; intra-node NVLink is roughly an order of magnitude better than PCIe, which is why placement matters so much. Replicated small ops — the layer norms / RMSNorm, residual adds, and embedding lookups — are typically computed redundantly on every rank rather than sharded, because they are cheap and replicating them avoids extra communication.
Bottom rows: memory and long context. The KV cache is sharded by head across the same ranks that own those heads, so the cache — often the dominant memory consumer for long sequences — divides naturally by TP degree, giving each GPU 1/N of the cache as well as 1/N of the weights. For very long contexts, sequence/context parallelism can layer on top, sharding the sequence dimension so no single rank holds the entire KV cache for a huge context. The ops strip is the practical reality: choosing the TP degree against the latency target, overlapping communication with computation where the framework allows, and placing ranks with topology awareness so every all-reduce stays on the fast fabric.
End-to-end flow
Trace one decode step for a model served with TP degree 4 across four GPUs on one NVLink-connected node. The model is generating token number 200 of a response; the KV cache for the prior 199 tokens is already resident, sharded by head so each GPU holds the keys and values for its quarter of the heads.
Input broadcast: the current token's hidden state (post-embedding) is present on all four ranks — embeddings are replicated, so every rank starts the layer with the same input vector. Attention: each rank runs its column-parallel QKV projection, producing Q, K, V for its own heads only; it appends the new K and V to its local shard of the KV cache; it computes attention scores and the weighted value sum for its heads entirely locally — no communication yet, because heads are independent. Then the row-parallel output projection: each rank produces a partial hidden vector, and the first all-reduce sums the four partials so all ranks hold the identical, complete attention output. Add the residual (replicated), apply RMSNorm (replicated).
MLP: each rank runs its column-parallel up/gate projection over its slice of the intermediate dimension, applies the SwiGLU activation locally, then the row-parallel down projection yields partial hidden vectors, and the second all-reduce sums them to the complete MLP output. Residual add, norm, and the layer is done. This exact dance — QKV local, all-reduce, MLP local, all-reduce — repeats for every one of the model's layers. After the final layer, the last hidden state feeds the (often column-parallel) LM head; an all-gather assembles the full logit vector, the sampler picks the next token, and it is broadcast to all ranks to begin the next decode step.
Now the performance reality this flow exposes. The compute per rank is 1/4 of the full model's per-token work, which is the win — but twice per layer, all four ranks must stop and synchronize on an all-reduce, and no rank proceeds until the slowest finishes. On NVLink this synchronization is fast enough that the compute savings dominate and per-token latency drops meaningfully versus one GPU. Move the same four ranks to two nodes connected by Ethernet and the picture inverts: each all-reduce now crosses the network, the per-layer synchronization cost swamps the compute savings, and decoding is slower than a single well-fed GPU would be. The identical algorithm is excellent or terrible purely as a function of where the ranks sit — which is why topology-aware placement is not an optimization but a correctness-of-design concern for TP.