SambaNova RDU

Reconfigurable dataflow versus thread-block execution

A GPU executes kernels: the driver launches a grid of blocks, each block is a cluster of threads, and threads run the same instruction stream on different data. Threads can cooperate via shared memory within a block and synchronise at block boundaries, but communication between blocks goes through HBM. A large model inference is a sequence of kernels, each one reading input activations from HBM, doing the work, and writing outputs back to HBM.

RDU inverts this. The compiler takes a dataflow graph — a DAG of operators connected by activation tensors — and statically maps it onto a spatial array of Processing and Compute Units (PCUs) with local memory between them. Once mapped, weights flow from one PCU’s memory to the next as needed, activations move directly from one compute unit to the next without visiting HBM, and execution is pipelined through the physical layout. There is no kernel launch, no thread synchronisation, no dynamic scheduling. The hardware is told: here is the shape of the computation; run it.

Why operator fusion becomes central rather than optional

On a GPU, an elementwise operation reads from HBM, computes, and writes back. A fused kernel reads once, does multiple operations in a single pass, and writes once. Fusion saves memory bandwidth and improves cache reuse; it is a valuable optimisation, but it is also a choice the compiler makes or a human writes by hand when the automatic fusion is not enough.

On RDU, fusion is not an optimisation; it is the default. The compiler fuses every operator it can into a single dataflow stage because there is no benefit to keeping them separate — they all run in the same spatial hardware slice anyway, and splitting them only means sending data in and out. The compiler is free to fuse across arbitrary boundaries because the execution is static and ahead-of-time. The result is that RDU compiler transformations look like manually optimised kernels that a GPU framework would have to hand-write, re-tune for each model shape, and recompile if the tile size or batch size changed.

Ahead-of-time compilation and the cost of dynamic shapes

GPU execution is dynamic: you launch a kernel, threads see the actual tensor shape at runtime, and control flow branches are taken. RDU is static: the graph is fixed, the mapping onto hardware is fixed, and the weights and activations flow along predetermined paths. This is a strength when the shape is known in advance and a weakness when it is not.

For inference with a fixed prompt length and batch size — the common case — this is not a limitation, and it is often an advantage. The compiler can specialise the entire computation to that shape. For a model where sequence length varies, batch size varies, or there is a lot of conditional logic, RDU requires either recompilation for each new shape or a runtime system that can adapt the execution. The latter exists but adds overhead; the former is not acceptable for real-time serving. This is not a deal-breaker for language model inference, where prompt length is usually known and batch size is tuned per deployment, but it is a constraint to know about.

Three-tier memory and weight streaming

RDU systems typically include on-package SRAM, on-package DRAM, and external DRAM. Weights live in external DRAM; activations and intermediate results live in on-package DRAM and SRAM. The PCU array is sized to maximize reuse of activations within the on-package tier while minimizing external bandwidth.

This changes the model-size constraint. On a GPU, inference batch size is limited by the on-package memory holding both weights and activations; you scale by adding more GPUs and using tensor parallelism. On RDU, weights stream from external DRAM according to the dataflow schedule, so model capacity is determined by external DRAM, not on-package HBM. A single node can inference models whose weight size fits in the DRAM tier, which is orders of magnitude larger than what would fit on-package. The trade-off is that weight streaming adds external memory bandwidth to the arithmetic count; you get larger models at the cost of higher memory traffic, which is often a favorable trade-off for serving.

Compiler as the abstraction layer

On a GPU, you write CUDA kernels or use a framework that emits them. The framework owns the kernel library and tuning. You control the algorithm. On RDU, the compiler is the abstraction layer. You express the computation in a high-level graph language (PyTorch, ONNX, or SambaNova’s own IR), and the compiler maps it to hardware.

The upside is that the compiler owns fusion, memory scheduling, and spatial mapping. You do not write custom kernels. The downside is that if the compiler does not know how to handle your operator, or if your operator is faster in a hand-written form on a different backend, you are stuck. Compiler maturity and operator coverage become the constraint. This is similar to XLA or TVM, but the hardware is not a GPU and the stakes are higher: a missing operator means the model does not run on this hardware at all, not just slower.

Ecosystem constraints and the vendor-specific path

RDU is a closed ecosystem. The model compilation pipeline is vendor-supplied. There is no open-source analog to CUDA or ROCm; there is no Triton fallback. If a framework or library does not ship a SambaNova-targeted compiler, you are paying an engineering team to build a bridge between your code and the RDU compiler.

PyTorch support exists, and ONNX bridges work for many models, but custom layers, quantisation formats, and attention variants that exist in the CUDA ecosystem sometimes have no RDU equivalent. The mitigation is to work with SambaNova’s support team to add operators or to rearchitect the model to use only covered primitives. This is not unusual for specialized hardware, but it is a real cost that does not appear in benchmark comparisons. Budget for integration and operator coverage, not just inference throughput.

Inference latency and throughput trade-off

GPUs are latency-optimised by default: inference runs fast because the hardware is fast. Raising the batch size improves throughput at the cost of latency. RDU optimises for throughput per node by pipelining the dataflow. A model that takes 10ms latency to infer on a single example may take 10ms total to infer a batch of 32 examples if the pipeline is deep enough to hide the latency. The throughput is remarkable; the latency per example is not.

For applications that need low per-request latency — chat, real-time translation, interactive search — high batch size and pipelined inference is incompatible. For applications that can batch requests — processing corpora, batch recommendation scoring, bulk inference overnight — RDU is a throughput win. Know which you need before evaluating.

Single-node deployment and on-prem niche

The architecture is optimised for large models on single nodes. Multi-node inference on RDU is possible but less compelling than the single-node case because you lose the weight-streaming advantage once you shard the model across multiple nodes. The natural deployment is: one model, one node, handle all the inference requests you can fit in a batch queue.

This is particularly attractive in enterprise on-prem settings. A single appliance running a foundation model or fine-tuned specialist model, managing request queues, serving many applications over a single inference point. The operational simplicity of not managing a cluster of GPUs, the licensing simplicity of a single vendor stack, and the security simplicity of inference staying on hardware not shared with other workloads all align. This is exactly the deployment model that makes single-node economics matter and multi-GPU tensor parallelism irrelevant.

Comparison with GPUs: a framework for thinking about it

Neither RDU nor GPUs are uniformly superior; they are different tools optimising for different objectives.

When to evaluate RDU: the right decision framework

RDU makes sense when: your model fits in a single node’s DRAM tier, you can afford to batch requests, you have a fixed prompt length and batch size, and your deployment is on-prem or single-tenant cloud. It makes sense when throughput per accelerator is the metric, not latency per request, and when your workload justifies a vendor integration cost for custom operators.

RDU makes less sense when: you need low per-request latency, you have variable sequence lengths or batch sizes, you require a wide ecosystem of supported layers and frameworks, or your model is so large that tensor-parallel GPU clusters are already your architecture. The honest evaluation is to pilot the specific model and workload, measure end-to-end, compute cost per million tokens, and price the vendor integration engineering. Compare not against peak numbers but against your actual serving topology and operational footprint.

Practical path to production and risk

Deploying on RDU means adopting a compiler stack that does not have the ecosystem depth of CUDA. Model compilation is deterministic once the shape is fixed, but the first shape you try may require operator coverage work or graph rewriting. Observability is vendor-supplied; your metrics, dashboards, and alerts are built on a smaller set of profiling tools than NVIDIA’s. Support for orchestration — Kubernetes device plugins, model serving frameworks, request scheduling — is more limited.

Mitigation: run a pilot with a well-defined exit criterion and representative traffic. Measure latency and throughput with your model and batch pattern. Check that the serving framework you use (vLLM, Text Generation WebUI, or custom) has RDU support or requires only shallow integration. Plan for the vendor support timeline if operators are missing. The risk is not technical but operational: deploying infrastructure that is smaller and less widely deployed than the incumbent requires higher confidence in the fit and lower tolerance for surprises at scale. Evaluate accordingly.