Latitude.sh is a bare metal GPU cloud provider focused on predictable pricing and physical servers for training and inference workloads. Unlike shared-tenancy hyperscaler instances, you rent entire machines with direct GPU access and control over the hardware stack. The model trades breadth of managed services for cost predictability and physical topology visibility. This piece covers the bare metal thesis, how allocation and pricing differ from on-demand cloud, what you inherit and what you lose, and the diligence questions that separate a bare metal provider from the hyperscaler default.
The bare metal thesis — from virtual multiplexing to physical control
The hyperscaler model is built around statistical multiplexing: thousands of unrelated tenants packed onto shared hardware, their demand peaks staggered so a small number of physical machines serve many more logical instances. That works for web services, databases, and other workloads with bursty traffic. It does not work for model training.
Large-model training is one enormous, tightly-coupled job spanning hundreds or thousands of GPUs running synchronously for weeks. It does not benefit from statistical multiplexing because it wants all the capacity all the time. Splitting a training run across scattered instances on a shared hyperscaler network adds latency, packet loss, and contention on oversubscribed interconnects. Bare metal providers invert the model: instead of one large cloud trying to be good at everything, they optimise the entire stack for one workload shape — single-tenant, large-scale, tightly-coupled compute — and let you rent whole machines with direct GPU access and full visibility into the hardware topology. The bet is that for sustained, high-utilisation compute, this specialisation delivers better economics than sharing.
Allocation shape — reserved blocks, not scattered instances
How capacity is handed to you determines what you can do with it. Hyperscalers allocate on-demand instances: you ask for machines, the scheduler finds whatever is available in your region, and you get back a list of disconnected servers. This is ideal for stateless services where machines are interchangeable and location does not matter. For a training cluster, it is useless. Sixty-four GPUs scattered across a datacentre with arbitrary network paths between them are not a cluster; they are sixty-four isolated GPUs paying the latency tax of random routing.
Bare metal providers allocate in blocks: a contiguous, topology-aware set of servers on the same fabric, reserved for your job and held for the duration. A block is the atomic unit. You ask for 8 A100 GPUs on the same NVSwitch island, the provider gives you exactly that, or tells you to wait. Capacity is planned, not summoned. Scaling is stepwise, not smooth — you double your block size in a step, not by incrementally adding instances. The question you ask is not 'what is the instance price' but 'what block sizes fit on a single non-blocking fabric domain, when, and with what SLA?'
Pricing structure — commitment replaces consumption
Because capacity is physically reserved and dedicated rather than statistically shared, the commercial model shifts from consumption to commitment. A hyperscaler GPU bill is granular: you rent instances by the hour, pay for what you use, and can stop or scale down within minutes. The contract is flexible but the effective unit cost is high because nobody is betting on steady utilisation.
A bare metal provider inverts this. Meaningful access typically means a term contract for a defined quantity of capacity — say, eight H100s for six months, not a credit card and an API call. Longer terms and larger blocks buy better unit pricing and better allocation guarantees. On-demand or short-term access, where offered, is the expensive exception rather than the default. The unit price is lower precisely because you are betting you will use it. The cost lever shifts from consumption to utilisation: you pay for reserved capacity whether or not your jobs fully occupy it. That makes cluster packing, job scheduling, checkpoint efficiency, and queue discipline first-order cost concerns rather than operational niceties. Wasted capacity costs you the same as wasted time.
Thin virtualisation and topology visibility
A hyperscaler GPU instance is a virtual machine: a hypervisor sits between your code and the silicon, mediating device access, enforcing tenant isolation on the network interface, and hiding the physical topology behind an abstraction. That is correct for a multi-tenant cloud. For a training job, it costs you three things: a slice of throughput, visibility into which GPU sits on which PCIe root complex and NUMA node, and the ability to tune the machine itself.
Bare metal providers lean toward bare metal or deliberately thin virtualisation. Your process runs directly on the hardware or through a hypervisor thin enough not to matter — passthrough devices, no resource limits, minimal overhead. You see the real topology. You know which GPU is on which socket, which socket has which NUMA memory, and which devices share a PCIe root. You can pin processes to cores, set environment variables for NCCL and MPI with real meaning, and profile without a hypervisor blurring the picture. The cost is inherited responsibility: driver versions, firmware, kernel tuning, and node health are at least partly yours now. The platform does not hide complexity; it pushes it to you.
Network fabric — the infrastructure bet
The interconnect is the single largest technical differentiator. Distributed training is dominated by collective operations — all-reduce over gradients, all-gather and reduce-scatter under sharded optimisers, continuous all-to-all traffic under tensor parallelism. A collective runs at the speed of its slowest link. On a hyperscaler, that link is a TCP stack over Ethernet on an oversubscribed spine designed for bursty, mostly north-south traffic. Packet loss, buffer overflow, and retransmit timeouts are endemic at cluster scale.
Bare metal providers build high-performance dedicated fabrics, typically InfiniBand, arranged as fat trees with low or no oversubscription at leaf and spine layers. The bisection bandwidth approaches 100 percent of the theoretical maximum. GPUDirect RDMA lets a NIC read and write GPU memory without a host bounce. NCCL is tuned against the real topology, not a generic default. The advantage is measurable: collective operations complete orders of magnitude faster, which compounds across millions of iterations. The trade is that the fabric is capital intensive, must be engineered before the racks arrive, and cannot be retrofitted. That is precisely why it is the moat.
Storage for training — throughput at full cluster width
A training cluster starves without a storage tier built for it, and this is where general-purpose object storage quietly fails. Data loading is thousands of workers issuing small random reads across a shared dataset; checkpointing is the opposite — an enormous synchronised burst of writes from every GPU at once, during which the accelerators are idle. A hyperscaler S3-like service handles neither pattern well at cluster scale. Reads hit per-prefix rate limits; writes hang waiting for acknowledgement; and egress bandwidth to move the checkpoint off the cluster becomes the bottleneck.
Bare metal providers pair the fabric with high-performance parallel filesystems, usually NVMe-backed, presented as a POSIX shared namespace and reachable over the same RDMA fabric. GPUDirect Storage, where supported, moves data by DMA from NVMe directly into GPU memory. Evaluate this tier on three metrics: read throughput at full cluster width (the number that matters), small-file metadata performance (how fast can you list a dataset), and how long a full-cluster checkpoint actually takes. That last number is pure wasted GPU time on every save — a factor that hyperscaler pricing quietly assumes away.
What you lose against a hyperscaler — breadth and operational maturity
Bare metal specialisation is a genuine trade, and the losses are real. The biggest is managed-service breadth. A hyperscaler offers hundreds of services — relational databases, NoSQL stores, data warehouses, streaming, serverless functions, analytics. A bare metal provider typically offers compute. Everything else — your application tier, your data platform, your observability stack — either stays on the hyperscaler or you run it yourself. If you are anchored to a hyperscaler by data gravity or compliance, the cost of the cross-cloud data path may eat the fabric savings.
Regional coverage is narrower: a handful of datacentres, not dozens of regions worldwide. Integrated identity and networking is thinner: fine-grained IAM, VPC constructs, private-link plumbing, and org-wide policy tooling live with the hyperscaler. And operational maturity in the unglamorous sense — incident history, compliance attestations, quota machinery, support depth, and ecosystem tooling that assumes your provider — sits with the incumbents. A bare metal provider at scale is still newer, with less public incident data and a smaller support organisation than AWS, Azure, or GCP.
The diligence checklist — what to ask before signing
The questions that actually separate bare metal providers are unglamorous and specific. Do not take numbers as read; you are renting a cluster, not an instance, and the cluster behaviour is what you will live with.
| Ask about | What you are really testing |
|---|---|
| Fabric topology | Oversubscription at leaf and spine, largest single non-blocking domain, measured all-reduce bandwidth at your block size |
| Block allocation | What sizes are available, whether they honour topology, time-to-provision, whether blocks can be grown or only replaced |
| SLA coverage | What counts as an outage, whether a degraded node (one GPU down) is covered, how a partially failed cluster is credited |
| Eviction and preemption | Whether reserved capacity can be reclaimed, notice period, what happens to your running job |
| Node replacement | Time to diagnose and replace a dead GPU or NIC, whether replacement happens in-place or you must wait for a new allocation |
| Storage throughput | Read bandwidth at full cluster width, metadata performance, real measured time for a full-cluster checkpoint |
| Egress cost | The hidden tax of moving checkpoints and data out of the cluster, especially to a hyperscaler data platform |
Get a burn-in window on the real block before the term starts. Run your own collective benchmarks, stress the storage tier, and do a multi-day soak. Marketing describes the hardware; only measurement describes the cluster.
When bare metal wins — workload shape and time horizon
The honest decision rule comes down to two dimensions: workload shape and time horizon. A bare metal provider earns its keep when you are running large, tightly-coupled, long-duration training on a known quantity of capacity, when fabric quality and sustained availability dominate your cost per useful GPU-hour, and when you can commit capacity far enough ahead to plan rather than summon. A 500-GPU continuous training run for six months on a dedicated block is the sweet spot. Sporadic runs, small-scale inference, or exploratory work are not.
A hyperscaler remains the better answer when your GPU use is bursty and small, when the accelerators live next to a large data footprint you are not moving, when regional coverage or compliance is the binding constraint, or when you value breadth of services more than the last increment of interconnect performance. Many large organisations settle on both — committed bare metal capacity for production training runs, hyperscaler infrastructure for the application and data tier, experimentation, and inference — and then spend their engineering effort on the seam between them, because that seam is where the cost and the failures accumulate.
The operational reality — health checking and cluster reliability
Hyperscaler GPU fleets fail in quantifiable ways: ECC errors, PCI link flaps, thermal throttling, firmware hangs. At single-instance scale, you retry or kill the job. At cluster scale, any single failure stalls the entire collective. A bare metal provider's operational surface is therefore all the difference. The best ones ship node health checking, automated draining of degraded hardware, fabric-level monitoring with real topology awareness, and job-restart integration so a checkpoint-restarting job lands on a fresh block automatically. The worst ones make you notice the failure, phone support, and wait for a technician.
This is the least visible and most consequential difference between providers: not who has GPUs, but whose cluster keeps a two-week training job alive. Ask for their incident history on multi-GPU failures. Ask whether node health checks are automatic or manual. Ask how a job in progress is handled when a node fails — is it migrated to another block, does it get killed and credited, or does it hang? The answers to these questions matter more than the headline specs.