SemiAnalysis Just Field-Tested Our Whitepaper. It Passed. Most Neoclouds Didn't.

Earlier this year Hedgehog wrote a detailed whitepaper explaining exactly why host and container-level isolation can't hold on GPU infrastructure. SemiAnalysis's latest newsletter dives deep into neocloud security, and they were not impressed. They spent 4 months testing 32 GPU clusters across 25 providers and showed that these failures aren't theoretical. They're happening widely, right now.

While it's gratifying to see independent validation, reading SemiAnalysis's findings is an uncomfortable experience, as nearly every failure they documented is a specific instance of a general failure mode our Secure Multitenancy whitepaper predicted from first principles. The industry's most rigorous cluster testing operation is discovering, in production, exactly the architecture problem we built Hedgehog to solve.

As attention shifts toward AI security, from discussions around Mythos to analyses of the Hugging Face incident, it is becoming increasingly clear that anyone building AI infrastructure must think more deeply about security. Until now, in the great AI build-out, speed has trumped almost everything else, but whether you are a neocloud, sovereign cloud or even an enterprise, you have to consider security across every stage of your cluster’s lifecycle.

What Semi Analysis found

Our central argument is that the standard isolation mechanisms such as VMs, containers don't work for GPU clusters. VMs need the IOMMU disabled for GPU performance, which breaks memory isolation. Containers can't accommodate tenants needing different kernels or drivers, and container escapes are a mature, ongoing category of vulnerability, not a solved problem. That leaves the network as the strongest remaining isolation boundary.

Shared Kubernetes control planes. One provider deployed shared K8s control-plane components across tenants using vCluster — directly against vCluster's own documentation, which specifies private nodes, not shared ones. The result: visible tenant metadata, two-year-out-of-date software, no default-deny NetworkPolicy, a publicly routable kubelet, and ultimately a working cross-tenant RCE proof-of-concept. Our whitepaper explains exactly why this happens structurally: Kubernetes CRDs are non-namespaced, global resources, so a genuinely multi-tenant cluster needs either multiple Kubernetes instances (requiring the virtualization that isn't a safe option here) or isolation that lives outside the host entirely — in the network. SemiAnalysis found the exact failure that occurs when a provider skips that step.

Container escapes. SemiAnalysis reproduced NVIDIAscape (CVE-2025-23266) — a year-old, publicly documented container toolkit vulnerability — against live provider infrastructure and successfully escaped a Docker container to root on the underlying host VM. We highlighted this risk as the reason container-only isolation is a standing liability, not a hardened boundary.

Manual network configuration failing exactly where we said it would. Our core technical claim is that BGP/EVPN, VXLAN, VRFs, route maps, and ACLs configured by hand across dozens of switches are "error-prone, difficult to audit, and slow." SemiAnalysis found this directly: providers with InfiniBand security keys (P_Key, M_Key, SA_Key) left misconfigured, in one case leaving 532 hostnames and endpoints visible on the fabric to a single tenant running saquery. In a second case, a provider's nodes held membership in both an isolated partition and the default partition simultaneously — a configuration error that broke both isolation and normal operation at once, and one that only a systematic, automated approach would reliably catch before it reached production.

Shrinking the blast radius

In another example, a single Prometheus API key with god-level read access turned one misconfigured dashboard into full visibility across every tenant on that infrastructure — GPU utilization, NVLink bandwidth, Slurm project names, vLLM inference stats, and log data belonging to AI research institutions, banks, telcos, and a national intelligence agency, all exposed through one shared monitoring layer.

In Our Security Math analysis, a flat-fabric, perimeter-only architecture carries a weighted blast radius of roughly 72% of the cluster per material security incident, versus approximately 4% for a fabric with hardware-enforced VPC isolation at the switch level. On a 1,024-GPU B200 cluster, our model estimated that architectural difference is worth approximately $7.0 million a year in avoided incident cost and recovered EBITDA — not because VPC isolation stops every attack, but because it bounds how far any single successful attack can travel.

AI Ops: Right Tool, Right Layer

SemiAnalysis's researchers found that frontier closed models (Opus, Fable) frequently refused to assist with legitimate, authorized security testing, pushing them toward more permissive but less capable open models — Kimi K3, GLM-5.2, DeepSeek V4 — to actually build proof-of-concept exploits and, in Hugging Face's case, to root-cause a real attack in progress.

That's a different problem than the one our whitepaper addresses, but the parallels are real. In both cases, the instinct to point an LLM directly at security-critical work runs into a hard limit, and using a smarter model won’t help. You need to put the right layer of tooling in the right place. Our whitepaper makes the same argument about network configuration specifically: an LLM generating switch config directly for a multi-tenant cluster produces diffs across 20-30 devices too complex for meaningful human review, with no guarantee a hallucinated route-map entry or missing ACL doesn't leak tenant data. The only viable fix is a deterministic Fabric Controller that takes small, human-reviewable intent (a VPC manifest, a peering declaration) and produces the same correct, tested switch configuration every time. AI generates intent, humans validate it, and a proven, deterministic system enforces it.

What to fix and how Hedgehog helps

SemiAnalysis’s closing recommendations boil down to:

• Fixing Bad Designs
• Stop doing namespace isolation on shared Kubernetes nodes.
• Stop relying on container-only isolation between tenants.
• Stop giving tenants access to BMCs and DPU management interfaces.
• Update software to minimum/latest versions
• Implement InfiniBand security keys correctly, every time, not just at initial commissioning.

Notice what most of these recommendations have in common: they ask operators to be perfect at manual, host-level configuration. So perfect for every commissioning, every key rotation, every node, forever. That's the real finding of the report. Isolation that lives on the host only holds as long as nobody makes a mistake, and across 32 clusters SemiAnalysis found that somebody always does.

Enforcing isolation in the fabric takes the mistake off the table. Define a tenant boundary once — a VPC, a peering, a Gateway, expressed as Kubernetes CRDs — and a controller compiles it into tested switch configuration that comes out the same, correct way every time, with continuous reconciliation catching drift before a tenant ever notices. And when a host-level control does fail, the fabric boundary is what keeps that failure inside one tenant instead of exposing all of them.

The core lesson from SemiAnalysis's testing is simple: isolation that depends on perfect host-level configuration will eventually fail. Isolation enforced by the network can survive those failures. If you're a neocloud operator reading the SemiAnalysis newsletter and recognizing your own infrastructure in it, the Hedgehog AI Network Planner will model exactly what closing that gap is worth for your specific cluster size — and the Secure Multitenancy whitepaper walks through the architecture in full technical depth, VPC by VPC, CRD by CRD.