Hallucination or Leak? The 24-Millisecond Question Hyperscalers Won't Answer
Anthropic is probably right that the Claude Code “Minecraft temple” report is a hallucination — but the physics of shared KV caches make the industry's next real leak a matter of when, not if.
By Elias Wong
Sparked by Potential session/cache leakage between workspace instances or consumer accounts · discussion
A Claude Code bug report landed this week alleging cross-tenant session leakage: mid-session, the agent began asking an enterprise user “what kind of bricks I wanted for my Minecraft temple.” Anthropic's team responded that they are “confident this is a hallucination” while taking the report seriously — and given that the reporter's own workspace contained Minecraft artifacts, they are probably right. But watch what the discussion did next: while Hacker News debated prompt injection, context contamination, and proxy routing bugs, almost everyone ignored the brutal physical constraints of how Key-Value (KV) caches are managed on multi-tenant GPUs — constraints that make the question “hallucination or leak?” permanently, structurally hard to answer. The fundamental reality is that every single token generated by these inference engines is pulled directly from the physical HBM3e memory cells assigned to them by the scheduler. Your sensitive enterprise codebase is resting completely unscrubbed in physical VRAM, and the only thing protecting it from being vomited into the next user's terminal is a highly fragile, logically-defined software pointer.
To understand exactly how this hardware bleeds data, we must look at the baseline mechanics of high-throughput LLM serving. The software engineering behind modern inference engines like vLLM is undeniably brilliant. Prior to these architectures, multi-tenant batch inference suffered from abysmal Model FLOP Utilization (MFU). Because the KV cache—the working memory the model uses to track context across tokens for long sequences—grows dynamically during the decode phase, inference engines historically had to pre-allocate massive, contiguous blocks of VRAM for every single user request. This caused catastrophic memory fragmentation, trapping usable FLOPS behind a wall of unusable, chopped-up SRAM and HBM allocations.
The breakthrough was PagedAttention. This architecture slices the KV cache into fixed-size logical blocks, managed like an OS virtual memory system. By separating logical memory from physical memory, the inference engine can dynamically map contiguous software addresses to highly fragmented physical HBM blocks scattered across the silicon die. The MFU gains are staggering, allowing hyperscalers to pack dozens of concurrent users onto a single physical GPU and continuously feed the tensor cores (deploying maximum batch sizes) rather than leaving an astronomically expensive piece of silicon idle. We completely concede that without PagedAttention, the unit economics of commercial LLM APIs would instantly collapse.
However, PagedAttention introduces a fatal physical bottleneck when deployed across mutually untrusted enterprise tenants. Let us trace a token through the actual hardware.
Imagine an 80GB H100 processing a massive proprietary codebase via a RAG pipeline for Tenant A. The inference engine allocates physical memory pages in the HBM3e stack to hold Tenant A's massive KV cache. When Tenant A finishes their session, the inference server simply updates its page table, deleting the logical software pointer.
At the raw silicon level, absolutely nothing happens.
The physical memory blocks are never zero-scrubbed. Tenant A's raw, unencrypted proprietary code remains baked into the physical memory cells waiting to be overwritten. When Tenant B initiates a prompt milliseconds later, the vLLM scheduler eagerly assigns those exact same "freed" physical blocks to Tenant B's new session. The hyperscaler relies entirely on the inference engine's software bounds-checker to ensure Tenant B's model weights cannot physically read past their logically allocated tokens.
When that logical software boundary inevitably breaks—whether through driver bugs, race conditions, or unhandled exceptions in the attention mechanism—you get a full-blown cross-tenant data leak. We have already seen the physical precedent of the LeftoverLocals vulnerability, where attackers reliably dumped local memory from shared GPUs by exploiting uninitialized physical registers and unscrubbed memory allocations. If an incident like Claude Code's ever turns out to be real, this is exactly the pathology it will be: a model reading dirty, unzeroed pages left behind in the memory array. The Minecraft temple was probably a hallucination; the attack class is not.
[Diagram Opportunity: A spatial layout of a physical H100 die and its surrounding HBM3e stacks, showing 'dirty' unzeroed KV cache pages being reassigned to a new tenant's pointer, highlighting the exact boundary where software isolation fails.]
Why do hyperscalers explicitly refuse to zero-scrub freed memory blocks before reassigning them? The answer is microscopic latency greed and brutal datacenter unit economics.
Let us do the math on the actual physical cost of zeroing an Nvidia H100. An H100 PCIe features 80GB of VRAM and boasts 3.35 TB/s of memory bandwidth. To calculate the raw latency penalty of completely scrubbing the entire 80GB chip, we divide the total memory capacity by the bandwidth (80GB / 3350GB/s).
The result is a negligible ~24 milliseconds.
Wiping a single 16MB KV cache block takes fractions of a single microsecond. Normalizing this physical security overhead against the standard Time-to-First-Token (TTFT) latency for an enterprise API call—which routinely hovers between 300 and 800 milliseconds due to network routing, prefill compute, and RDMA overhead—reveals that scrubbing the physical memory would add less than a 5% latency penalty in the absolute worst-case scenario.
Yet hyperscalers are actively choosing to bypass this step. When operating a 100,000 GPU cluster drawing 150MW of datacenter capacity, hyperscalers will mercilessly cull any microscopic delay to prevent their gross margins from compressing. They refuse to inject even a 24-millisecond scrub cycle because in a massive distributed inference pipeline utilizing tensor parallelism across NVLink domains, a 24ms delay on a single node can cascade into massive tail latencies for the entire batch. In the brutal war for API market share, infrastructure operators are optimizing exclusively for maximum throughput and TTFT, treating physical data isolation as an acceptable casualty. They operate massively parallel, multi-tenant infrastructure under the delusion that logical isolation is equivalent to physical isolation.
This hardware reality fundamentally breaks the prevailing enterprise AI narrative. Corporate Chief Information Security Officers are signing massive contracts for Enterprise Zero Data Retention (ZDR) policies, assuming their sensitive data is protected. ZDR on shared, unscrubbed multi-tenant infrastructure is a mathematical lie.
If logical boundaries inevitably fail—and LeftoverLocals proved they can—enterprises must demand true single-tenant physical hardware isolation. We can model the exact financial bloodbath this requires.
Forcing strictly isolated VRAM allocation physically starves the compute cores, completely destroying the unit economics of the API. If a hyperscaler is forced to dedicate a physical GPU strictly to one tenant to guarantee hardware-level ZDR, they lose the ability to multiplex idle compute cycles across dozens of smaller requests. Back-of-envelope against public pricing, running an 8x H100 node without PagedAttention batching drops continuous utilization from a highly optimized ~65% down to sub-15%.
[Diagram Opportunity: A TCO waterfall chart contrasting the negligible latency cost of zero-scrubbing (~24ms) against the catastrophic Capex penalty (3x-4x TCO increase) required to achieve true single-tenant hardware isolation.]
To maintain the exact same token throughput and service levels while enforcing single-tenant physical isolation, hyperscalers would have to radically over-provision their datacenter footprint. The math is unforgiving: dedicating a fully populated 8x H100 OAM baseboard to a single enterprise tenant to guarantee physical isolation forces that single customer to absorb the entire amortization schedule of the asset. We are talking about a fundamental shift in the billing model. To grasp this massive financial gulf, we must tear down the underlying capex requirements for the baseboard, the NVSwitches, the networking transceivers, and the cooling infrastructure required to host a node that is suddenly vastly underutilized.
Instead of subsidizing a single GPU node across hundreds of concurrent API calls, the enterprise must swallow the entire cost of the $300,000 server, the 400G InfiniBand ConnectX-7 NICs, the Top-of-Rack (ToR) switches, the highly expensive 800G optical transceivers, the power distribution units, and the facility's liquid cooling infrastructure. You cannot simply divide the rack cost by fifty anymore; the unit economics revert to the dark ages of dedicated hardware provisioning. We calculate this throughput penalty effectively multiplies inference Capital Expenditure (Capex) and Total Cost of Ownership (TCO) by a factor of 3x to 4x.
The economics of modern AI inference are built on a precarious foundation of shared physical memory and fragile logical pointers. Hyperscalers have designed their entire datacenter footprint around maximizing MFU through multi-tenant batching, silently pushing the severe physical security risks onto their customers while pocketing the massive gross margins.
Enterprise ZDR on shared infrastructure cannot exist under these constraints. If enterprises demand true physical data isolation to protect their proprietary IP, they must immediately accept the catastrophic Capex penalty of single-tenant deployments, fundamentally repricing the cost of AI access across the entire industry. If hyperscalers keep optimizing solely for MFU on unscrubbed PagedAttention, a real cross-tenant bleed is a matter of when, not if — and when it comes, “we're confident this is a hallucination” will be a much harder sentence to type. The brutal physics of the silicon dictate that you can have microscopic latency optimizations, or you can have guaranteed physical data security. You cannot have both without paying the single-tenant toll.