Podcast - How Open Compute Is Shaping the Future of AI Clusters

When the Open Compute Project Foundation (OCP) gathered for its Global Summit last year, more than 11,000 people showed up, putting a once-niche hardware community within striking distance of NVIDIA’s GTC. That kind of growth says a lot about where AI infrastructure is heading, and about who is building it.

On our second episode of The AI Hedge, I sat down with James Kelly, Vice President of Market Intelligence and Innovation at OCP, to talk through how open standards are helping the industry keep pace with AI’s relentless demands.

From Servers to the Whole Data Center

James traced OCP back to 2011, when Meta, then Facebook, joined Intel and a handful of others to open source hardware designs for the data center. Meta was buying servers by the hundreds of thousands and wanted to avoid being locked into vendor forklift upgrades. So, it designed its own servers and needed somewhere to steward those open designs.

Fifteen years later, the scope has widened dramatically. As James put it, the community now writes specifications “from grid to chip or even concrete to chip,” covering power, cooling, networking, storage, and even chiplet design. A roughly 20-person foundation coordinates thousands of engineers, and you do not need to be a member to take part. “Part of its success is that it’s free and open for anybody to come and collaborate,” he said.

According to its tagline, OCP aims to offer ‘community driven hyperscale innovation for all.’ Specifications adopted by hyperscalers stay open for everyone else, from neo-clouds down to enterprises and smaller cloud and SaaS companies. It is a mission Hedgehog knows well, as our goal has always been to help customers ‘network like a hyperscaler’.

Why Disaggregated Inference Changes the Hardware

The conversation moved to disaggregated inference, one of the most talked about topics in AI infrastructure right now. Disaggregated inference refers to running prefill and decode on separate, specialized machines rather than on the same GPU. Since each has different performance characteristics, it makes sense to match different hardware to each one to deliver better performance.

According to James, as AI workloads become more specialized, we're likely to see greater diversity inside AI data centers, with multiple generations of GPUs and specialized accelerators working together to deliver performance and efficiency. He pointed to lingering demand for older GPUs, NVIDIA’s move to acquire Groq, and Cerebras’ recent IPO as signs that heterogeneity is here to stay.

Open Clusters and the Networking Glue

To prevent teams from reinventing the wheel, OCP’s Open Cluster Design initiative has published specifications for building blocks called pods, complete with rack counts, cabling, and accelerator layouts that multiple vendors can slot into. As James noted, “there’s no point in designing a snowflake for every cluster.”

But mixed hardware only works if it can talk. When accelerators from different vendors share memory across a network, proprietary, siloed stacks become a dead end. “You want your network to ubiquitously connect up whatever you decide to deploy next,” James said. This thinking underpins our recent contribution of two AI networking reference architectures, one for training and one for inference, which sit alongside OpenAI’s host-driven multipath reliable connection (MRC) specification for training clusters.

What Comes Next

Looking ahead, James expects continued adoption of double-wide ‘open rack wide’ systems, virtual reality (VR) racks, advances in liquid cooling, higher-voltage power delivery, and a brand new optical circuit switching project. A recent low-voltage DC (LVDC) contribution was one of the most collaborative contributions OCP had received, drawing some 70 companies and hundreds of contributors.

Conclusion

OCP’s story is a useful reminder that the AI buildout is not only a race for chips, but also a race for shared standards. As accelerators multiply and clusters grow more heterogeneous, the organizations that come out ahead will be the ones that can mix and match hardware without rebuilding everything each time a new generation lands. James’ central message, that openness and interoperability beat isolated silos, is the kind of foundation a fast-moving market needs. And going by the growing interest in OCP events, the open infrastructure community clearly agrees.

Listen to the full episode.