Fixing upload bufferbloat on pfSense: 130 ms to 27 ms
My upload added up to half a second of lag whenever something in the house was uploading. One FQ-CoDel limiter on pfSense brought loaded latency back to idle.
My connection tested at about 930 Mbps down, and everything felt fast, until something in the house started uploading. Then video calls stuttered, games rubber-banded and SSH sessions typed in slow motion. The download number was never the problem. The upload queue was.
This is bufferbloat: when a link is saturated, packets pile up in an oversized buffer (usually in the cable modem), and every other packet waits in line behind them. Throughput looks fine on a speed test. Latency does not.
Measure first
I ran the Ookla CLI from a wired server, because it reports latency while the link is loaded, not just idle ping. Two runs against different servers told the same story:
| Speed | Typical latency under load | Worst spike | |
|---|---|---|---|
| Idle | – | 24 ms | 27 ms |
| Download | 930 Mbps | 37 ms | ~330 ms |
| Upload | 37–44 Mbps | 87–134 ms | ~520 ms |
Download was mildly bloated. Upload was bad: half a second of added delay for anything sharing the link with a big upload.
I also sampled the firewall's WAN counters once a second during an upload test. The modem happily accepted bursts close to 60 Mbps, but the line only sustained about 40–45. That gap is exactly where the queue builds up.
The fix: own the queue
The trick with bufferbloat is to make sure the queue forms somewhere you control, and then manage it well. On pfSense that means a limiter set slightly below the real upload rate, using the FQ-CoDel scheduler:
- Limiter just under the sustained rate, so the modem's buffer never fills.
- FQ-CoDel splits traffic into per-flow queues (so one big upload can't starve a video call) and drops early from any flow whose queue sits too long.
- A floating "match" rule on WAN, outbound, sends traffic through it.
I shaped upload only. Download bloat was mild, and shaping a gigabit link in software costs real CPU for very little gain.
Three gotchas
1. The GUI defaults can be zero. pfSense fills FQ-CoDel's target and interval from kernel sysctls. Before the dummynet module has ever loaded, those read as 0. I set them explicitly to the standard values (5 ms target, 100 ms interval) rather than trusting the defaults.
2. Leave ICMP alone. The rule matches TCP and UDP only. Gateway monitoring pings, and the firewall pair's failover heartbeats, stay out of the limiter, so a busy upload can never make the WAN look "down" and trigger a failover.
3. The rule reload is asynchronous. Changing the limiter and reloading returns immediately; the new rate applies a few seconds later. My first "test at the new setting" was really still the old one. Now I check that the limiter reports the new rate before measuring anything.
Tuning the rate
I started at 40 Mbps, about 90% of what the line sustains, and re-tested:
| Upload under full load | Before | Limiter at 40 |
|---|---|---|
| Typical latency | 87–134 ms | 26–28 ms |
| Worst spike | ~520 ms | 36–76 ms |
Loaded latency is now essentially the same as idle. Then I tried 45 Mbps to win back some throughput. On some runs the bloat came straight back (73 ms typical, 347 ms spikes), because 45 is above what the line reliably sustains. So it stays at 40. Giving up a few Mbps of peak upload for a connection that never lags is an easy trade.
Because the firewalls run as a failover pair with config sync, the limiter and its rule copied to the standby automatically, so shaping survives a failover too.
Takeaways
- Test latency under load, not just speed. A speed test that only reports Mbps hides this entirely.
- Find the sustained upload rate, not the burst, and set the limiter just below it.
- Shape the direction that hurts. Here that was upload only.
- If the ISP ever raises the upload rate, re-measure and move the limiter up.