Optimal Opus

Why Does One Epoll Worker Take All the Load?

EPOLLEXCLUSIVE, as in: exclusively worker zero

10 min read

Last time, we dug through the LKML archives and learned that every long-term flaw of Epoll was pointed out before it shipped. A fun history lesson, but history lessons don’t come with error bars. So I went and built a benchmark harness to measure how Epoll’s threading models actually behave under load.

One of those benchmarks returned a number so wrong-looking that I spent an evening chasing it through the kernel’s wait queues. The answer turned out to be a real, documented behavior of EPOLLEXCLUSIVE, made worse by a benchmarking sin I didn’t even know I was committing.

The title is an homage to Cloudflare’s excellent Why does one NGINX worker take all the load?, which I only fully appreciated after independently rediscovering its conclusion the hard way.

The Contenders

The harness pits five event-loop architectures against the same fixed workload: clients connect over loopback and collectively write 1 GiB in 1 KiB payloads, while the server’s job is simply to drain every byte. Total work stays constant while client fanout varies (256 → 512 → 1024), so we’re measuring how each architecture distributes work, not how much work there is. The machine is a 32-core Linux box, and every multi-threaded model gets 128 workers.

  1. single_thread - one thread, one epoll. The C10K classic.
  2. shared - one epoll instance, 128 workers all calling epoll_wait on it.
  3. shared_oneshot - same, but every fd is armed with EPOLLONESHOT and re-armed after each drain.
  4. exclusive - 128 workers, each with its own epoll instance, all watching the same listener with EPOLLEXCLUSIVE. Accepted connections stay with the worker that accepted them.
  5. per_worker - 128 workers, each with its own epoll and its own listener via SO_REUSEPORT. The kernel hashes incoming connections across them. This is roughly what modern NGINX does.

Each model runs both level-triggered and edge-triggered. Criterion does the timing. Here are the means (level-triggered shown; edge differs only where noted):

model c256 c512 c1024
single_thread 169 ms 170 ms 175 ms
shared 109 ms 89 ms 68 ms
shared (edge) 35 ms 41 ms 44 ms
shared_oneshot 53 ms 61 ms 68 ms
exclusive 111 ms 126 ms 130 ms
per_worker 32 ms 34 ms 42 ms

Most of these rows behave exactly the way the theory says they should:

  • single_thread is dead flat at ~170 ms because one core draining 1 GiB is the bottleneck no matter how many clients you spread it over. Level vs edge are identical, since draining every fd to EAGAIN makes the trigger semantics operationally equivalent.
  • per_worker wins, and of course it wins: no shared epoll to contend on, no fd ever visible to two threads, and the kernel’s 4-tuple hash spreads connections evenly. There’s a reason the industry converged here.
  • shared + edge-triggered lands within ~10% of per_worker. An edge event gets queued once and consumed by exactly one of the 128 waiters - a natural work queue. The gap is the lock contention on the single shared epoll instance.
  • shared_oneshot pays ~1.6× over plain edge-triggered. That’s the price of one extra epoll_ctl(EPOLL_CTL_MOD) syscall per event, serialized through the same epoll lock. Level and edge are statistically identical here, because once every event disarms the fd, trigger semantics barely matter.
  • shared + level-triggered is the known pathology: every ready fd wakes many of the 128 waiters, who stampede into reads that mostly return EAGAIN. The tell is in the scaling - it gets faster with more clients (109 → 68 ms), because more distinct fds spread the herd thinner. When adding load improves your throughput, you’re not measuring I/O, you’re measuring contention.

Which leaves us, dear reader, with one row that makes no sense at all.

The Anomaly

EPOLLEXCLUSIVE exists to fix the thundering herd on accept. Instead of waking every worker watching the listener, the kernel wakes one. Each of my 128 workers has a private epoll, every connection lives in exactly one of them - architecturally this should be breathing down per_worker’s neck.

It benchmarks at 111–147 ms - four times slower than per_worker, and barely better than a single thread.

And the suspicious part isn’t even the mean - it’s the variance. Per-sample spread for the healthy multi-threaded models is 10–35%, which is what honest scheduler noise looks like with 1100 threads on 32 cores. The exclusive model’s spread is 1–5%, as tight as the single-threaded run. It’s slow the same way the single_thread model is slow: reproducibly.

That smelled less like overhead and more like the parallelism simply wasn’t there.

Interrogating the Benchmark

My first instinct was to rebuild the exact event loop in a standalone binary, but with per-worker byte counters bolted on. Same accept loop, same drain policy, same 128 workers, same client pool, same generation-based timing.

At steady state it ran in 38 ms, with 56 workers participating. The standalone replica is three and a half times faster than the benchmark of the very same code.

At this point you should be hearing the Kill Bill sirens. When a faithful replica refuses to reproduce your measurement, either the replica is unfaithful or your benchmark is measuring something you didn’t intend.

Re-running the full criterion suite reproduced the slow numbers perfectly - all six exclusive configs landed between 110 and 147 ms, while per_worker clocked its usual 32 ms seconds later in the same process. So the effect is real, stable, and specific to the exclusive model. Time to instrument the real thing instead of a replica. I had each worker report its connection count and bytes drained at shutdown:

DIAG worker 1: conns=154 mb=55549.4
DIAG worker 0: conns=77  mb=27774.7
DIAG worker 2: conns=19  mb=6853.5
DIAG worker 4: conns=3   mb=1082.1
DIAG worker 5: conns=2   mb=721.4
DIAG worker 3: conns=1   mb=360.7

256 connections. 128 workers. Six of them ever touched a byte, and two of them owned 90% of the traffic. At c1024 it was worse: the top worker held 680 of 1024 connections. My gloriously parallel 128-worker server was, functionally, a two-thread server with 126 spectators.

And one more detail that turns out to be the whole story: it was always workers 0 through 3. The first threads spawned. Every config, every run.

The Mechanism

EPOLLEXCLUSIVE does not round-robin. When the listener becomes readable, the kernel walks the listener’s wait queue and wakes the first exclusive waiter it finds - and position in that queue is fixed by registration order. Worker 0 registered first, so worker 0 gets the wakeup. The manpage is honest about this, in its way:

When a wakeup event occurs and multiple epoll file descriptors are attached to the same target file using EPOLLEXCLUSIVE, one or more of the epoll file descriptors will receive an event with epoll_wait(2).

“One or more will receive an event.” No promise about which, and certainly no promise about fairness.

Now combine that with what every event loop tutorial (and my server) does on a listener event:

loop {
    match listener.accept() {
        Ok((stream, _)) => { /* register with my epoll, keep it forever */ }
        Err(e) if e.kind() == WouldBlock => break,
        Err(_) => break,
    }
}

Accept-until-EAGAIN means one wakeup doesn’t win you one connection - it wins you the entire backlog. During a burst of 256 near-simultaneous connects, worker 0 gets woken, inhales everything queued, and goes back to sleep at the head of the queue, ready to do it again. Workers 1–3 pick up scraps from the moments worker 0 was busy. Workers 4–127 never get woken up at all.

And because connections in this benchmark are accepted once per config, before the timed region, that one lopsided dice roll gets frozen and replayed for every sample. The “benchmark” of a 128-worker architecture was a measurement of one worker draining ~65% of a GiB at single-threaded pace, forever. That explains the mystery mean, and it explains the suspiciously tight variance too.

The Heisenbenchmark Twist

One loose end: why did my standalone replica spread connections across 56 workers when the real benchmark concentrated them onto 2?

I chased some wrong theories first - maybe glibc’s cached thread stacks made client threads spawn faster in a long-lived process (nope, no effect), maybe extreme skew alone explained the time (nope: artificially forcing all connections through the backlog only got me to 54 ms). The answer was simpler and dumber. Watch what happens when the same binary runs the same config five times in a row, in one process:

config run 0: last-gen  39.6 ms | active workers 58 | top conns [20, 19, 17, ...]
config run 1: last-gen 125.8 ms | active workers  4 | top conns [165, 67, 19, 5]
config run 2: last-gen 118.6 ms | active workers  4 | top conns [157, 75, 21, 3]
config run 3: last-gen 137.2 ms | active workers  3 | top conns [183, 60, 13]
config run 4: last-gen 127.6 ms | active workers  4 | top conns [174, 63, 17, 2]

A cold process is healthy, and a warm process is pathological - same code, same machine, seconds apart.

In a freshly-exec’d process, the client threads are slow off the line - page faults, lazy linking, cold everything - so connections trickle in, each wakeup finds one or two queued, and the herd spreads across dozens of workers. In a warm process the connect burst is fast enough to pile the backlog deep while worker 0 is still busy registering its last batch, and the accept loop does the rest. A criterion suite is always warm by the second benchmark. My replica was always cold, because it was always run number one.

The race that decides your load balance is over in the first few milliseconds of the connection storm, and which way it goes depends on how warmed-up your load generator is. My benchmark harness wasn’t just observing the system - it was participating in it.

What I’m Taking Away From This

EPOLLEXCLUSIVE is a thundering-herd fix, not a load balancer. It guarantees you fewer wakeups, but it makes no promise about who gets them. If connection-to-worker assignment is sticky (and in the own-epoll-per-worker design, it is), wait-queue ordering plus greedy accept loops will starve most of your workers. SO_REUSEPORT won by 4× in my data and it’ll win in yours, which is why NGINX ended up there. If you must use EPOLLEXCLUSIVE, bound your accepts per wakeup so other workers get a turn.

A benchmark that freezes accept-time state measures a single dice roll. My harness establishes connections once per config to keep setup out of the timed region - a perfectly reasonable choice that quietly turned “throughput of this architecture” into “throughput of the unluckiest accept distribution I happened to roll.” If work assignment happens at accept time, report the per-worker load distribution next to the throughput number, or the throughput number is fiction.

Tight variance is a smell. The healthiest-looking row in my results table - beautiful 1% spreads and all - was the most broken one. Variance that’s too low for the amount of concurrency you allegedly have means the concurrency isn’t happening.

The repo with the harness, the diagnostic reproducers, and the fixes is up on my Gitea. The kernel, as usual, was working exactly as documented - I just hadn’t really read the documentation until a benchmark called me a liar.

If You Liked this, You Should Also See…