Vera Rubin scales up the rack and now the network needs to scale out as well

SemiAnalysis's recent analysis of Vera Rubin vs Grace Blackwell inference is, as usual, very thorough, and worth reading in full. These early results from Coreweave’s implementation show, depending on the exact date of the baseline, NVIDIA has improved between 4X and 10X tokens/sec/user. Regardless of what starting point you want to pick, Rubin is a massive improvement in rack scale inference performance and economics.

Coreweave ran these tests on a pre-production rack with no scale-out fabric attached and to quote SemiAnalysis this setup "uses wide EP and PD disagg, which uses the NVL72 scale-up backplane and proves that it is working well." The benchmark data shows the scale up domain is performing at the current leading edge and NVLink made 72 GPUs behave as a single accelerator. Given the ramp up issues for GB200 NVL72 Oberon, this once again shows how well NVIDIA is executing.

However, in production systems traffic will cross the rack boundary and how the wider network performs will decide the overall cost and performance outcomes. The long-context, multi-turn, mainly agentic traffic that increasingly dominates production will add the following challenges:

• Prefill and decode pools across racks
• KV-cache transfer over RDMA
• Scale-out collectives
• Storage traffic
• Multi-tenancy congestion
• Increased operational overhead

NVIDIA designed Rubin as a two-network system

NVLink 6 scales the system up, and ConnectX-9 with Spectrum-X Ethernet scales it out. NVIDIA's own Dynamo documentation draws the same line for disaggregated serving — when prefill and decode workers share a scale-up domain, the KV cache can move over NVLink; when they're placed across nodes or racks, NIXL transfers it over an RDMA-capable scale-out network, and Dynamo routes requests among worker pools that the orchestration layer scales independently as demand shifts.

The type of work being done will of course impact the performance outcome. Quick questions into a chat app are most likely to be served entirely within one scale-up domain. Agentic coding assistants will have a very different profile. Those load tens of thousands of tokens of repository context in a compute-heavy prefill, and then generate a comparatively short, memory-bound decode.

At larger scales, those prefill and decode pools are sized and scaled independently — a service carrying thousands of developers needs a very different prefill:decode ratio at 9am than at 2am — and once those pools span racks, the KV-cache transfer and worker coordination ride scale-out Ethernet, not the backplane inside either pool. When capacity spreads across buildings or regions for power, latency, or data-residency reasons, the same requirement extends further.

Scale out decides the bill

SemiAnalysis had previously introduced the concept of “One Chart to Rule them All” and updated it with the latest benchmarks and the improved rack-scale performance on Vera Rubin makes these racks 3 to 5 times cheaper on a token/sec/user basis. The performance of the scale-out fabric determines how much of that improvement operators can actually collect. Congestion between pools, retries on an unreliable link, a misconfigured switch, config drift between sites — each one idles GPUs that have already been paid for.

Disaggregation exists because splitting compute-bound prefill from memory-bound decode and scaling each independently is cheaper than serving both from identical, tightly coupled nodes. But that efficiency gain only shows up in the bill if the network connecting the pools is as automated and reliable as the backplane inside each one.

Where Hedgehog fits

Hedgehog doesn't replace NVLink or UALink, and we have no interest in competing with the scale-up interconnects or the switch vendors building on the open UALink standard. We operate the Ethernet fabric around those scale-up domains: a declarative, automated fabric for deploying, connecting, and operating GPU capacity across racks and tenants.

Hedgehog supports Spectrum-X and aligns with NVIDIA's Cloud Partner reference architecture, and also supports open OCP and SONiC-based designs, so an open scale-up standard gets an equally open scale-out layer next to it.

Once a production AI factory’s traffic crosses the scale out network (almost every one eventually does), the performance and reliability of that network determines how much of Rubin’s rack-level performance becomes billable capacity.

We've written about the mechanics of disaggregated inference in detail and the importance of ROCE so if you're planning a multi-rack inference deployment and want to pressure-test the scale-out side of your cost model, reach out — we're happy to help.