BitTorrent for LLMs, Coordination, and the Speed of Light

Every few weeks, a thread predictably surfaces on Hacker News where someone makes the same argument around decentralized AI. The premise usually goes like this: we don't need giant, power-hungry datacenters to run massive 405-billion parameter models, because all we really need is a BitTorrent for LLMs. I understand the human romance of decentralization, but trying to port the physics of file-sharing to generative inference is structurally misguided. BitTorrent works phenomenally well precisely because downloading a file is an embarrassingly parallel task that actively avoids centralized coordination. Inference demands the exact opposite. Let's grab a whiteboard, look at the actual physics of distributed generation, and run the math.

I was reading through the documentation for the Petals project recently, and they state the reality of the architecture plainly, noting that autoregressive generation requires sequential computation. This is the unyielding physical constraint of the entire pipeline. In a heavily sharded setup, token $N+1$ physically cannot be computed until token $N$ has traversed the entire network of nodes. You cannot download tensor chunks out of order. You cannot parallelize a strict dependency graph.

Let's model a theoretical "happy path" single forward pass to isolate our core variable. Imagine we take a massive LLM and sequentially shard it across eight consumer nodes over a wide area network. Bandwidth is the usual scapegoat in these internet debates, but bandwidth is entirely fine. If we look at the raw config.json for a 405B parameter model, we find a hidden_size of 16,384. At bfloat16 precision, the tensor activation payload passing between nodes is barely 32 kilobytes.[^1] Your home internet connection can easily handle 32KB.

The true bottleneck is strict coordination latency. In optical fiber, light moves at roughly 200,000 kilometers per second. If our eight consumer nodes are spread randomly across North America, the absolute physical floor for transmission—assuming an extraordinarily optimistic 30 milliseconds of routing delay per hop over seven WAN hops—is 210 milliseconds per token. That gives us a strict, speed-of-light bounded theoretical maximum of roughly four tokens per second.

That is lethargic. But survivable.

But that is the pristine academic theory (the frictionless vacuum where TCP window limits and BGP route flaps don't exist). In reality, a sequential pipeline deployed over consumer internet isn't just transmitting math, it is surviving a relentless barrage of dropped packets, thermal noise, and mundane routing failures. We need to introduce the actual cosmic rays of consumer networking, specifically bufferbloat caused by a roommate streaming video while a background operating system update pegs the router's queue depth.

To see how this breaks the system, I was running the math on a bimodal latency distribution, drawing heavily on Dean and Barroso's classic paper The Tail at Scale. Let's restrict our simulation to a single escalation of load—tail latency compounding over sequential hops.

Assume our eight-node pipeline is mostly healthy. Each individual node has only a 1% probability of experiencing a 200-millisecond latency spike due to local bufferbloat or jitter. If this were BitTorrent, a 1% failure rate means 99% of your independent chunks arrive instantly, and the lone straggler finishes a moment later with zero impact on the overall cluster throughput.

Because autoregressive inference is strictly sequential, the mathematical escalation is ruthless. The probability that a single token generation step completes without hitting that 200ms penalty anywhere in the chain is $0.99^8$, which equals roughly $0.923$. This means nearly 8% of your single-token generations will hit the tail latency. Now escalate the load to a standard generation response of 100 tokens. The probability of navigating that 100-token sequence without a crippling network delay drops to $0.923^{100}$—or roughly $0.0003$. At a mere 1% tail latency risk per node, a standard 100-token response has a 99.97% probability of encountering massive, compounding pipeline stalls.

Oof!

When an edge node gets stalled by this tail latency, it sits idle, holding onto GPU memory, waiting for the preceding node to finish its work. The system state becomes deeply pessimistic (I’m assuming the node ahead of me is currently drowning in TCP retransmits...), and because the nodes are entirely decoupled administratively, there is no centralized control plane to shed load, drain queues, or reroute the tensor in real-time. In a synchronized pipeline, a delay at hop number four immediately exhausts the queue depth of hop number three, which inevitably halts hop number two. The entire distributed cluster drops to the speed of the slowest temporarily degraded node. When that localized queue depth finally hits its limit, it isn't just a mathematical limit being breached; it is a timeout error surfacing on a user's screen halfway across the globe, generated by a fragile daisy-chain of anonymous machines that nobody is actually on-call to fix.

Hyperscale datacenters are fundamentally engines of geographic compression. We pack GPU racks into massive concrete facilities specifically to eliminate this exact coordination latency, keeping the hops on specialized, deterministic interconnects that bypass TCP/IP congestion algorithms entirely. Attempting to force an $O(N)$ sequential dependency chain across geographically uncoordinated WAN nodes is solving the problem at the wrong abstraction layer. You are asking uncoordinated consumer hardware to absorb the operational realities of a distributed system without giving it the dedicated network topology required to survive it.

BitTorrent scales because it avoids coordination, but autoregressive inference bottlenecks because it demands it, leaving us with an unyielding heuristic of distributed engineering: at scale, coordination is just latency by another name.

[^1]: A single vector of 16,384 dimensions, assuming a 16-bit float (2 bytes), comes out to exactly 32,768 bytes. The math doesn't care about architectural vibes.