For a decade, “GPU compute” and “CUDA” were the same sentence. AMD Instinct accelerators running ROCm are the first alternative that shows up in production LLM serving rather than in a research footnote — not because the silicon suddenly got good, but because the software stack crossed the threshold where mainstream frameworks work out of the box. That changes the question. It is no longer “can this hardware run my model?” but “which parts of my stack are secretly CUDA-shaped, and what will it cost to unshape them?” This article walks the programming-model mapping, the porting cliffs tooling cannot smooth over, the ecosystem gaps that decide outcomes, and a way to evaluate the switch that survives contact with your own workload.
HIP: a near one-to-one analogue of the CUDA runtime
The foundation of the ROCm story is HIP, the Heterogeneous-compute Interface for Portability. HIP is deliberately not a new idea — it is CUDA with the prefixes changed. cudaMalloc becomes hipMalloc, cudaMemcpyAsync becomes hipMemcpyAsync, cudaStream_t becomes hipStream_t. Kernels are still marked __global__, still launched with triple-angle-bracket syntax, still index themselves with threadIdx and blockIdx, and still declare on-chip scratch with __shared__.
The conceptual model carries across intact: a grid of thread blocks resident on a compute unit, a memory hierarchy of registers, on-chip scratchpad, cache, and HBM, occupancy limited by registers and scratchpad per block, coalescing rules that reward contiguous access. If you know how to reason about a CUDA kernel’s memory traffic and occupancy, that knowledge transfers directly. HIP also compiles back to NVIDIA as a thin pass-through layer, so a HIP codebase can be a genuinely portable single source rather than a one-way migration.
Where the abstraction is not one-to-one
The gaps are small in number and large in consequence. The most important is the execution width: NVIDIA schedules threads in warps of 32, while AMD’s datacenter architectures execute wavefronts of 64. Any kernel that hardcodes 32 — a shuffle-based reduction unrolled by hand, a lane-mask computed as a 32-bit integer, a __ballot result stored in an unsigned int — is silently wrong rather than loudly broken. Ballot masks are 64-bit on AMD, and warpSize must be read, never assumed.
Terminology shifts too: the streaming multiprocessor becomes a compute unit, shared memory is the Local Data Share, and matrix math is issued through AMD’s matrix-core instructions rather than NVIDIA’s mma family. Cooperative-group semantics and anything depending on the precise timing of implicit warp synchronisation need review. These are exactly the places where automated translation produces code that compiles and returns the wrong numbers.
What hipify does, and what it quietly leaves behind
AMD ships hipify-perl and hipify-clang to mechanise the rename. The Perl version is textual substitution — fast, dependency-free, and happy to mangle a string literal that happens to contain the word cuda. The Clang version parses the translation unit properly, so it understands types and macros, but it needs a working CUDA installation and a build that actually compiles.
Either way, hipify translates API surface, not meaning. It will not fix a hardcoded warp width. It cannot touch inline PTX assembly, because there is no mechanical mapping from one instruction set to another. It does not know that your build system passes NVIDIA-specific architecture flags, that your CMake logic keys off CUDA_FOUND, or that a dependency you pull in is itself unported. Treat hipify as a tool that eliminates the boring 90% of the diff and leaves the 10% that needs judgement — that 10% is the entire schedule risk.
Where a port actually costs: kernels, not applications
Ordinary application code ports almost for free. The expensive surfaces are narrow and predictable. Hand-tuned kernels are the worst case: a kernel tuned for a specific tile shape, register budget, scratchpad size, and warp count is tuned for a machine that no longer exists on the other side. It compiles and runs, but the tuning is now noise, and re-tuning is an engineering project rather than a build flag.
Inline PTX must be rewritten against AMD’s ISA or replaced with portable intrinsics, and whoever wrote it did so because the intrinsics were not good enough. Template metaprogramming libraries built around NVIDIA matrix instruction shapes — the CUTLASS-style layer under many fast attention and GEMM implementations — have an AMD counterpart in Composable Kernel, but counterpart is not drop-in. Budget by counting custom kernels in your dependency tree, not lines of code.
Library maturity is the real gap, not raw hardware
The persistent misreading of this competition is that it is about silicon. AMD has shipped credible datacenter parts for several generations; what took longer was the layer of tuned libraries that turns a peak number into delivered throughput. The correspondence is broad but the maturity is uneven:
| CUDA side | ROCm counterpart | Practical note |
|---|---|---|
| cuBLAS / cuBLASLt | rocBLAS / hipBLASLt | Core GEMM paths are solid; tuning varies by shape |
| cuDNN | MIOpen | Matters most for convolutional models |
| CUTLASS | Composable Kernel, rocWMMA | Different templates, different tiling |
| NCCL | RCCL | API-compatible; tuning and topology differ |
| Thrust / CUB | rocThrust / hipCUB | Usually a clean rename |
| Nsight, cuda-gdb | rocprof, rocgdb | Capable, but a different workflow |
The gap that bites is rarely a missing library. It is a specific fused kernel, quantisation format, or attention variant that exists on one side and has a slower generic fallback on the other.
Framework operators with no ROCm path
PyTorch is the good news: the ROCm build is upstream, and it deliberately keeps the torch.cuda namespace and the cuda device string, so most model code runs unmodified. HIP masquerading as CUDA is a compatibility decision, not sloppiness — the enormous body of code that writes .to("cuda") simply works.
The breakage lives one layer down, in packages shipping their own compiled CUDA extensions: bespoke quantisation kernels, specialised attention implementations, sampling or MoE-routing kernels, and vendor-specific inference compilers with no AMD analogue. Some have ROCm forks, some lag upstream by a release or two, some have nothing. The structural mitigation is Triton: kernels written in Triton, including those that torch.compile generates for you, retarget to AMD without a rewrite. The more of your fast path is Triton or standard ATen operators, the cheaper every future hardware decision becomes.
The memory-capacity advantage and what it unlocks
AMD’s consistent architectural bet has been more HBM capacity per package than the contemporary competing part. Capacity is not a benchmark headline, but it changes serving topology, which is where the money is. A model that fits in fewer devices needs a lower tensor-parallel degree, and tensor parallelism costs an all-reduce on every layer.
Halve the tensor-parallel degree and you shrink the per-layer collective; reach a single device and you delete it outright. The workloads that benefit most are the ones where capacity is the binding constraint: large dense models you would otherwise have to shard aggressively, long-context serving where the KV cache dominates and headroom converts directly into batch size, mixture-of-experts models with large aggregate weights, and single-node fine-tuning of jobs that would otherwise go multi-node. Higher batch at the same latency is a throughput win no kernel tweak matches.
Collectives and scaling out with RCCL
RCCL is AMD’s collectives library and it is deliberately NCCL-API-compatible: the same all-reduce, all-gather, reduce-scatter and broadcast primitives, the same communicator initialisation, largely the same environment variables. Distributed training and multi-GPU inference code written against NCCL does not need restructuring, and the algorithmic families — ring and tree reductions chosen by message size — are the same.
What differs is the fabric underneath. Intra-node, AMD systems connect accelerators over AMD’s own high-speed links rather than an NVSwitch, so the topology and therefore the achievable all-reduce profile is not identical, and tuning knobs that were folklore on one platform are not automatically right on the other. Inter-node you are back on RDMA over the same class of network. The practical rule: never carry over a tuned collective configuration on faith. Re-run your sweeps at your real message sizes before concluding anything about scaling.
What breaks when you move a running PyTorch or vLLM deployment
Framework compatibility is the part everyone tests. The operational layer is the part that surprises people. The host stack is different: a different kernel driver, a different userspace runtime, different container base images, and a version matrix between ROCm, the framework build, and the serving engine that is tighter than teams expect.
Below that, the plumbing renames itself. Device visibility is controlled by different environment variables. Kubernetes needs a different device plugin advertising a different resource name, so every manifest, quota, and scheduler rule that names the vendor needs editing. Telemetry is the sharpest edge: your exporter, dashboards, alert thresholds, and capacity models are built on NVIDIA-specific metrics and do not transfer, so you can be running fine and flying blind. GPU partitioning exists on both sides but with different granularity and tooling. And serving-engine feature flags — attention backend, quantisation format, graph capture, custom all-reduce — do not have identical support matrices, so the fastest configuration you painstakingly found will not be the fastest one there.
Evaluating the switch honestly
Vendor benchmarks are chosen; yours are not. The only evidence that means anything is your model, at your precision, context length and batch shape, measured end-to-end. Run a pilot with a written exit criterion, and measure cost per million tokens rather than price per accelerator-hour — capacity and communication savings show up in the former and are invisible in the latter.
Then price what does not appear in a throughput chart: the engineering weeks to port or replace custom kernels, the observability rebuild, the second CI matrix, the on-call team learning a new profiler, and the risk that one critical dependency lands late. Weigh those against real benefits — supply availability, negotiating leverage, and no single-vendor dependency in your cost structure. The strongest position is not picking a winner but staying portable: keep the fast path in Triton and standard operators, keep vendor-specific code behind a narrow interface, and the next hardware decision becomes an evaluation instead of a migration.