Why architecture matters here
Multi-LoRA serving is architectural because it resolves a tension between two things that pull in opposite directions: personalization, which wants a distinct model per tenant, and efficiency, which wants to share one model across everyone. Full fine-tuning gives you personalization at the cost of a full model copy per tenant; a single shared model gives you efficiency at the cost of no personalization. LoRA's low-rank decomposition is the wedge that lets you have both — the base is shared, the adapter is per-tenant and tiny — but capturing that benefit at serving time requires an architecture that can hold one base and many adapters and apply them selectively. Without that architecture, you still end up running separate models; with it, one deployment serves the whole fleet.
The defining constraint is GPU memory, and it explains most of the design. The base model consumes the bulk of GPU memory and is loaded once; each adapter is small, but a thousand adapters still add up, and GPU memory is the scarcest resource in the system. So the architecture treats adapters like a cache: a working set of hot adapters lives in GPU memory where they can be applied at full speed, while the long tail of cold adapters lives in cheaper host memory or on disk and is staged into the GPU on demand. This makes adapter management a caching problem — hit rate, eviction policy, staging latency — layered on top of the model-serving problem, and the quality of that cache largely determines whether the system feels fast or janky.
The second defining constraint is that batching must survive adapter diversity, and this is the technically hardest part. Efficient LLM inference depends on batching many sequences through the same big matrix multiplications; the base model preserves this because every request shares the base weights. The adapters are the complication: within one batch, request A might use adapter X and request B adapter Y, so the low-rank delta each request needs differs. The architecture handles this with specialized grouped kernels that, in a single launch, apply the correct adapter to each request's rows — effectively a batched, per-segment matrix multiply. The base pass is shared across the whole batch; only the small adapter contribution is per-request. This is what keeps a mixed-tenant batch nearly as efficient as a single-tenant one, and it is the innovation that makes multi-LoRA serving worthwhile rather than a fancy way of running requests serially.
There is a deeper architectural payoff that follows from sharing the base: everything else in the serving stack can be shared too. Because all requests run through the same base model, they share a single attention computation and a single, unified KV cache managed exactly as it would be for a base-only server — the adapters change the projection weights, not the fundamental shape of the forward pass. That means multi-LoRA serving composes cleanly with the other big serving optimizations: continuous batching, paged KV cache, and the rest all apply unchanged, with adapter application slotted in as an extra per-request term. The architecture's elegance is that it adds personalization as a thin, cache-managed layer on top of an otherwise-standard high-throughput serving engine, rather than requiring a different engine per tenant — which is exactly why it scales to thousands of fine-tunes on hardware that could hold only a handful of full models.
The architecture: every piece explained
The top row is the request path. Each incoming request names the adapter it wants — a tenant id, a task id, or an explicit adapter reference. The router resolves that name to a concrete adapter and decides how to schedule the request into a batch. The base weights are the large, frozen model loaded once and shared by every request regardless of adapter. The LoRA adapters are the small per-tenant low-rank deltas that, applied on top of the base, produce that tenant's specialized behavior. The essential shape is one big shared thing and many small specific things.
The middle row is memory and compute management. The adapter cache holds the hot working set of adapters in GPU memory so they can be applied immediately; the adapter store holds the cold long tail in host memory or on disk, from which adapters are staged when requested. The batched kernel is the specialized operation that applies different adapters to different requests within one batch — grouping the per-request low-rank multiplies so the batch runs as a single efficient launch rather than a loop over tenants. Underneath, the shared KV cache and attention run once per batch as in any base-model server, because the adapters modify projections, not the attention structure.
The third row holds the two policies that govern the cache. The cold-load path is what happens on a cache miss: the requested adapter is not resident, so it must be fetched from the store and staged into GPU memory before the request can run, adding latency to that first request. The eviction and fairness policy decides which adapters to keep and which to drop when GPU memory is full — typically a least-recently-used policy, but with per-tenant limits so that one heavy tenant cannot evict everyone else's adapters and monopolize the cache. These two policies together determine the tail-latency behavior of the whole system.
The ops strip names the metrics that tell you whether the architecture is healthy. Adapter cache hit rate governs how often requests pay the cold-load penalty; cold-load latency governs how bad that penalty is; per-tenant fairness governs whether the shared GPU is shared equitably; and the total adapter count governs whether the working set even fits. Watching these four is how you keep a multi-tenant deployment fast and fair rather than letting a few tenants or a bloated adapter library quietly degrade everyone's latency.
End-to-end flow
Follow a warm request. A tenant sends a prompt tagged with their adapter id. The router resolves the id, checks the adapter cache, and finds the adapter already resident in GPU memory. The request joins the current batch alongside other tenants' requests. The engine runs the shared base forward pass for the whole batch, and the batched adapter kernel adds each request's low-rank delta using its own adapter — the tenant's rows get their adapter, their neighbor's rows get a different one, all in the same launch. Attention and the KV cache run once for the batch. The tenant gets a response specialized by their fine-tune, at essentially base-model throughput, having shared almost all the compute with unrelated tenants. This is the design working as intended.
Now a cold request. A different tenant, idle for hours, sends a request for an adapter that has been evicted to the store. The router sees a cache miss and triggers the cold-load path: the adapter is fetched from host memory or disk and staged into GPU memory. Only once it is resident can the request run. That first request pays the staging latency; subsequent requests for the same adapter are warm and fast. Whether this cold penalty is a minor blip or a serious tail-latency problem depends on how large adapters are, how fast the staging path is, and how often the working set turns over — which is why cold-load latency and cache hit rate are first-class metrics, not afterthoughts.
Consider the pressure case: many active tenants, more adapters wanted than fit in GPU memory. The cache is now the battleground. A pure LRU policy would let whichever tenants are busiest keep their adapters resident and push everyone else to the cold path, which is efficient in aggregate but unfair — a bursty heavy tenant can evict a steady light tenant's adapter repeatedly, giving the light tenant terrible tail latency. Per-tenant limits fix this by capping how much of the cache any single tenant can occupy, guaranteeing each active tenant a foothold. The router and eviction policy together are effectively a scheduler for scarce GPU memory, and their tuning is where multi-tenant fairness is won or lost.
Finally, watch what happens as the adapter library grows without bound. The system was sized for a working set of a few hundred hot adapters, but the product succeeds and there are now tens of thousands of registered adapters, with a much larger and more diffuse active set. Cache hit rate drops, cold loads become common, and the staging path — host-to-GPU transfer, or worse, disk reads — turns into the bottleneck. The forward pass is no longer the constraint; adapter movement is. The remedies are architectural: shard tenants across replicas so each replica's working set stays small enough to cache, speed up the staging path, or place adapters in faster storage tiers. The key operational insight is that multi-LoRA serving shifts the bottleneck from compute to adapter memory management as the tenant count grows, so capacity planning must track adapter working-set size, not just request rate — a server that is fine at a hundred tenants can thrash at ten thousand even though its GPU compute is barely touched.