The Coordination Tax of Sequential Order
Enforcing sequential order in concurrent systems causes catastrophic memory bloat, which is why modern data engines make this coordination an explicit choice.
By Owen Tate
Sparked by Pre-Release of Polars 2.0 · discussion
It feels like every time a major data processing engine changes its default behaviors, the same underlying argument erupts across the internet. I was scrolling through the Hacker News thread regarding the Polars 2.0 release yesterday, and users were broadly bristling at the decision to drop default row ordering in streaming mode. The prevailing sentiment assumes sequential output should be functionally free. To our baseline intuition, a dataset fed into a pipeline from line 1 to 100 should simply exit that pipeline from line 1 to 100.
But physical computing reality dictates otherwise. Order is fundamentally a strict coordination protocol that masquerades as a natural law. In a single-threaded execution model, order feels completely free because time itself enforces the sequence for you. The CPU physically cannot process line 2 before line 1. Shift into a concurrent, multi-core reality, and enforcing that same sequence requires independent threads to continuously agree on the state of the universe before they are permitted to emit results. We are replacing the natural physics of time with artificial, programmatic barriers. And coordination—as any distributed systems text will rapidly remind you—is the sworn enemy of scalability.
Let’s run the math on a whiteboard to see exactly where this comforting abstraction leaks. Picture a modern 16-core CPU chewing through a massive CSV file. We divide the workload into independent, uniform chunks to maximize our hardware utilization. If every thread takes exactly 10ms to process a chunk, the system simply zips the outputs together perfectly at the end. For $N = 16$ concurrent threads, we achieve a beautiful, linearizable speedup. It’s an incredibly seductive mental model. The system behaves exactly as the single-threaded mind expects, scaling gracefully across the cores while magically maintaining strict sequence.
The catch is that perfect uniformity doesn't exist outside of textbooks. Those 16 threads are operating in a remarkably hostile microarchitectural environment. Let’s introduce a single, mathematically trivial 15ms page fault on Thread 2. Threads 3 through 16 finish their 10ms chunks right on schedule and are ready to flush their processed data downstream. Unfortunately, because the API implicitly promised the user perfectly sequential order, they are strictly forbidden from doing so. They must park their results in RAM.
To maintain this illusion of sequence, the streaming engine is forced to instantiate a hidden reassembly buffer under the hood. We can quantify this architectural trap explicitly using Little's Law, which dictates that the long-term average number of items in a stationary system ($L$) equals the long-term average effective arrival rate ($\lambda$) multiplied by the average time an item spends in the system ($W$).
As the wait time ($W$) for Thread 2’s delayed chunk spikes due to the OS interrupt, the arrival rate ($\lambda$) of completed chunks from the other 14 active threads remains highly optimized and constant. Consequently, the size of our reassembly buffer ($L$) balloons in direct proportion to the delay. If each thread is chewing through 50MB of data every 10ms, a minor stutter on a single core means gigabytes of parsed data are suddenly piling up in memory, waiting for one straggler to cross the finish line. We have successfully converted a CPU-bound task into a memory-management crisis. [^1]
And that is just a single fault. If Thread 14 hits a garbage collection pause right after Thread 2 recovers, the system oscillates. The engine spends progressively less time doing actual analytic work and more time desperately allocating massive, continuous blocks of memory to hold parked chunks. This is classic metastability. The system's attempt to recover from a transient delay (buffering until order is restored) generates structural overhead that makes future delays significantly more likely.
Stepping away from the queuing math, consider what this systemic decay looks like to the human tasked with keeping the pipeline running. When a container inevitably runs out of memory because of severe thread skew in a reassembly buffer, the resulting telemetry rarely points a clear finger at coordination constraints. The metrics dashboard simply shows a standard (and supposedly safe) group_by operation suffering an unexplained Out-Of-Memory kill. The operator is left debugging ghosts. They have no visibility into the hypervisor's thermal state or that initial 15ms page fault, leaving them entirely blind to the fact that completely independent compute threads were secretly coupled together by an implicit API promise.
This is why dropping the default ordering constraint in modern data engines is a brilliant piece of system design. By forcing developers to write an explicit flag like maintain_order=True, the engine shifts the cost of coordination from a hidden architectural landmine into a highly legible statement of authorial intent. The codebase itself now physically warns the human writing it: I am intentionally choosing to pay for a reassembly buffer here, and I accept the risk of catastrophic memory bloat under duress.
Good API design doesn't just manage hardware; it dictates operational behavior. Coordination is never actually free. As system architects, we can either bury the invoice for that coordination deep in the underlying implementation, or we can force the caller to read the price tag before they execute the code.
[^1]: If you've ever wondered why a beautifully optimized, heavily concurrent data pipeline suddenly thrashes into swap space on a supposedly idle machine with plenty of headroom, congratulations, you've formally met the reassembly buffer.