AWS Replaced 100,000 iptables Rules to Keep the Ledger Honest

Here's a bottleneck nobody sees in the serverless pitch: the ledger. On any Lambda worker the real constraint isn't the microVMs, the CPU, or the cooling — it's the record you have to keep of every packet that crosses the network, complete and correctly attributed, for workloads that live a few hundred milliseconds and then vanish. AWS engineers just published a deep dive on the system they tore out to fix it, and the answer is almost embarrassing. The old design counted packets with iptables. Each new Firecracker microVM added more rules to the chain, so a worker running a couple thousand microVMs needed well over a hundred thousand iptables rules just to keep the books — and iptables walks its rules roughly linearly, so every packet paid a tax that grew as the host got busier. You can't double your density when your audit log gets more expensive the more you pack in. The old kernel module had a second, quieter problem: it didn't speak IPv6. 'A record that can't see half the address space isn't one you can trust,' they wrote, and dual-stack support was effectively a death sentence for the approach.

The replacement is a small, boring, well-chosen set of pieces. eBPF programs sit on the traffic-control hook at each network's virtual devices and emit one compact ~24-byte event per packet into a ring buffer — they only watch, there's no code path that drops or rewrites anything. Rust taggers (one per network, a few hundred KB of RAM each) drain the buffers and roll per-packet events into per-flow records, and a single privileged orchestrator per host holds all the elevated permissions, handing open file descriptors to the unprivileged taggers over a Unix socket. Two details do most of the heavy lifting for real operators. First, the output is byte-for-byte identical to the old Amazon Ion format, so every downstream billing, flow-log, and metering consumer kept working without noticing the swap — and it gave them a record-for-record way to prove the new system saw everything. Second, the ring-buffer floor was derived from first principles (peak rate × drain interval × event size × two directions), not guessed, which is what lets them promise no dropped events under a burst. Together that buys roughly double the microVM density, constant-time lookups instead of a linear rule walk, and IPv6 as a first-class citizen — while shrinking the trusted privileged surface to one small process per host.

Source image 1

The shape of it matters more than the specifics, and it travels. Wherever you're running many tenants on a host — Kubernetes pods, edge runtimes, the sandboxes people are spinning up to run AI agents — the same four shapes hold: observe from outside the hot path, size your buffers from a real limit, separate tenants at the point of capture, and keep the bolt pattern when you swap the

Source image 2

engine. The uncomfortable part is that most of us have an iptables-shaped ledger somewhere: a bookkeeping job that gets more expensive the busier the box gets, which is exactly when the record is supposed to hold up. What's the one line item on your platform that quietly taxes every request, and what would you do the day it stopped scaling?

Sources

• How AWS Lambda logs every flow across thousands of microVMs per host with eBPF and Rust — The New Stack
• Firecracker — secure and fast microVMs for serverless computing
• Announcing the Firecracker Open Source Technology — AWS Open Source Blog