Testing a Reconnect Ladder: Assert the Climb, Not the End State
A realtime dashboard I worked on has a simple promise for when the connection drops: try to reconnect a few times with growing gaps, and if that doesn't work, stop pretending and fall back to polling. A small status chip tells the user which of those it is doing: live, reconnecting, or "live updates unavailable".
The reconnect schedule was pinned as a constant and unit-tested as a value. What nothing did was actually drop a live connection and watch the client go through it. When I wrote that test, I learned that the obvious version proves almost nothing, and that the honest version has two traps of its own.
Two broken clients that pass the obvious test
The obvious end-to-end test is: go offline, check the chip says "reconnecting", wait, check it eventually says "polling". It reads well. Now picture two broken clients.
- The quitter. Someone sets the retry count to zero. On disconnect the client shows "reconnecting" for a moment, gives up at once and switches to polling. The test sees "reconnecting", then sees "polling". Green.
- The optimist. Someone removes the attempt limit. socket.io's backoff doesn't stop on its own: by default it keeps retrying forever. The chip says "reconnecting" indefinitely. If your "eventually polling" check has a long timeout, or you only assert the first state, this one can slip through too. And the user sees a chip that is always one attempt away from clearing.
These are opposite bugs, and an end-state test is blind to both of them because the behaviour is the path, not the destination. "Retry three times with backoff, then give up" is a claim about a sequence over time. So the test has to look at the sequence.
Assert the climb
Here's the shape I ended up with. Each check rules out one specific broken client:
- Go offlinestart the clockcheck: chip says "reconnecting", not "polling"
- ≥1sAttempt 1failscheck: gap ≥ first delay
- ≥2sAttempt 2failscheck: gap ≥ second delay
- ≥4sAttempt 3failscheck: gap ≥ third delay
- Pollinggave up, on purposecheck: exactly 3 attempts, and it got here
| Client | End-state test | Climb test |
|---|---|---|
| Gives up at once | passes | fails: 0 attempts, too fast |
| Retries forever | can pass | fails: never reaches polling |
| Correct | passes | passes |
A few things make this possible.
1. Turn off jitter
Backoff with jitter is the right default in production: you don't want every client in a building reconnecting at the same instant after a blip. But jitter makes timing unassertable. socket.io randomises each delay by 50% unless you say otherwise. Set the randomisation factor to zero, and the schedule becomes something you can reason about in a test:
const socket = io(SERVER_URL, {
reconnectionAttempts: 3,
reconnectionDelay: 1000, // 1s, 2s, 4s with the default factor of 2
reconnectionDelayMax: 4000,
randomizationFactor: 0, // deterministic gaps
});
socket.io.on("reconnect_failed", () => setStatus("polling"));
socket.on("disconnect", () => setStatus("reconnecting"));If you want jitter in production, make it a config value and turn it off in the test build. Don't loosen the test until it tolerates jitter. At that point it can't tell 0 seconds from 1.
2. Make the client really offline
My first idea was to block the socket path with Playwright's request routing. That doesn't work, and it fails in a misleading way. Routing intercepts new requests. A WebSocket that is already open isn't a request any more, so it carries on as if nothing happened. The test then times out waiting for "reconnecting", and you start doubting the client.
The fix is to take the whole browser context offline, which does drop the existing socket:
await expect(chip).toHaveAttribute("data-state", "live");
const droppedAt = Date.now();
await context.setOffline(true);
try {
await expect(chip).toHaveAttribute("data-state", "reconnecting");
await expect(chip).toHaveAttribute("data-state", "polling", {
timeout: LADDER_BUDGET_MS,
});
const attempts = await page.evaluate(() => window.__reconnectLog);
expect(attempts).toHaveLength(3);
for (let i = 1; i < attempts.length; i++) {
expect(attempts[i] - attempts[i - 1]).toBeGreaterThanOrEqual(DELAYS[i] - SLACK_MS);
}
expect(Date.now() - droppedAt).toBeGreaterThan(DELAYS[0]);
} finally {
await context.setOffline(false);
}The attempt log is a tiny test-only hook: in test builds the client pushes a timestamp on every reconnect_attempt event. My first version only had the final check, "it took longer than the first delay", and that alone is enough to catch the quitter. Counting attempts and checking the gaps catches the next bug too: someone changing the factor or the count without meaning to.
The same trap works in the other direction. If you want a test to show that a feature breaks without the socket, block the socket before the page loads. Block it afterwards and the test passes anyway, which looks like proof the test is worthless. It isn't. The block just came too late.
3. Use lower bounds, and an upper bound that is just a budget
The lower bound is what does the work. Polling should only start after every attempt has failed, so reaching it sooner than the first delay means the ladder was skipped. The upper bound only has to rule out "never gives up", so it can be generous. Exact equality would make the test flaky for no benefit.
Green on Linux CI, red on my Mac
Then the test failed locally. On my Mac it timed out with the chip stuck on "reconnecting", never reaching "polling". The same commit passed on Linux CI, and it failed the same way on the main branch on my machine, so it wasn't my change. A rerun produced a second, scarier error: the test tried to go back online in its finally block after the browser context had already been torn down. That was only fallout from the timeout. I learned to read the first error and ignore the cleanup noise.
The cause, as far as I can tell: I had sized the upper bound from the schedule, the sum of the delays plus some margin. But the real duration of the ladder is the delays plus the time each attempt takes to fail. On Linux, offline emulation makes a connection attempt fail almost instantly. On macOS it doesn't seem to fail as fast, so each attempt eats a chunk of time before the next delay even starts. Three slow failures were enough to go over a budget that was comfortable on Linux. I haven't measured it precisely yet, so take the exact mechanism as my best explanation, not a proven one.
Whatever the details, the general lesson holds: "how long until the network says no" depends on the platform. Offline emulation, DNS, firewalls, and how quickly the OS refuses a connection all differ between a developer laptop and a CI container. Any test whose timing includes failed connection attempts needs to account for that. The options I'm weighing:
- Give each attempt a short connect timeout in the test build, so a failed attempt takes a known maximum time on every OS.
- Derive the budget from the schedule: sum of delays, plus attempts × per-attempt timeout, plus slack. Don't use a round number that happened to work on CI.
- Assert on the attempt log rather than wall-clock time wherever you can. Counts don't care how slow the OS is.
Trade-offs
This test is slow: it has to sit through the whole ladder. I keep the schedule short in tests and run it in a serial project so it doesn't fight other tests for the shared context. The test-only attempt log is a small concession to testability that I'd make again. Deterministic delays mean test builds behave a little differently from production, which is why jitter should be a config value and not a code path.
Takeaway
When the behaviour you care about is a sequence, an end-state assertion will pass for clients that skip the sequence entirely. Remove the randomness, cut the connection for real, and assert the climb: how many attempts, and how far apart.