Bug #9280
Updated by Adam Barnard 9 days ago
PROBLEM STATEMENT: In IPFW/divert IPS mode (-d PORT), the kernel queues diverted packets in the divert socket's receive buffer. When the buffer is full, divert_packet() frees the packet: sbappendaddr_locked() fails, then soroverflow_locked() and m_freem() run (sys/netinet/ip_divert.c, FreeBSD 14 and 15). Suricata therefore doesn't see the packet and it isn't counted in divert statistics. FreeBSD sets that buffer at fixed size of 65636 bytes (DIVRCVQ). The source has a /* XXX sysctl ? */ comment, and it isn't runtime-tunable. That's ~43 full-size packets, under a millisecond at 1 Gbit/s. A short stall of a receive thread drops packets, and a single fast TCP flow results in TCP flow rate back-off to far below the link rate. Suricata doesn't set SO_RCVBUF on these sockets. EVIDENCE: On OPNsense 26.7.7 (FreeBSD 15.1) with Suricata 8.0.7, pf divert-to, 4 listeners (a 4-core Intel N305 CPU) and TLS encryption-handling set to bypass: - A single-stream TLS download ran at 68 Mbit/s through the IPS and then at 275 Mbit/s when the divert rules were disabled. - Receive threads were at 16-48% CPU. - DTrace on soroverflow_locked showed 24-130 drops per 5 s, with stack divert_packet <- pf_test <- pf_check_out <- pfil_mbuf_fwd <- ip_output <- ip_forward. - netstat -anx -p divert showed R-HIWA 65636 on all four sockets. Reproduced in a FreeBSD 15.1 VM: pf divert-to into Suricata (workers), vnet jails, 20 ms emulated RTT, one iperf3 download, medians of 3 runs of 20 s: - no divert: about 1190 Mbit/s - default buffer: 17.4 Mbit/s, with 154-169 soroverflow_locked() calls from divert_packet() per run - SO_RCVBUF 4 MiB with kern.ipc.maxsockbuf 16 MiB: 1213 Mbit/s, with no overflows - ipfw divert rules instead of pf: 17.8 and 1216 Mbit/s Related reports: - The same issue reported by someone else: https://github.com/opnsense/ports/issues/298 - Jason Ish suggested this cause on #8377 (2026-09-08); that ticket turned out to have a different cause, however. VERIFIED ON THE ORIGINAL ROUTER: Same Opnsense box as above. Build: Suricata 8.0.7, OPNsense's port patch and this change, with ipfw.buffer-size set to 4mb and kern.ipc.maxsockbuf set to 16777216. - @netstat -anx netstat anx -p divert@: divert: all four sockets show R-HIWA 4194304 and R-BMAX 16777216. That includes the SO_REUSEPORT_LB listeners OPNsense adds. - The same single-stream TLS download through the IPS: 244 Mbit/s, against 68 Mbit/s before. The client's figures without the IPS were 262-275 Mbit/s, so what's left is mostly its Wi-Fi. - DTrace during that download: no soroverflow_locked() calls from divert_packet(), against 24-130 per 5s before. PROPOSAL: Add ipfw.buffer-size, named like af-packet's and pcap's buffer-size. Suricata would set SO_RCVBUF on each divert socket before bind(). - It accepts a size from 64kb to 64mb. A value of 0, or leaving it unset, keeps today’s behavior (the kernel default). - Errors log a warning and keep the kernel default. Failing the receive thread is avoided because divert fails closed. - Two things are limited in kern.ipc.maxsockbuf. The kernel refuses a buffer larger than 8/9 of its size, keeping the remaining 1/9 for mbuf overhead. It also caps the mbuf memory the buffer may hold, at min(8 * size, maxsockbuf), and a full-size packet takes about 1.7 times its length in mbufs, which means the warnings ask for kern.ipc.maxsockbuf of at least twice the buffer size. - The cap avoids a kernel overflow. The kernel keeps sb_mbmax in 32 bits and sbspace() reads it as an int. From 256 MiB, with kern.ipc.maxsockbuf at 2 GiB or more, it can exceed INT_MAX; at 512 MiB it wraps to 0. Either way the socket drops every packet. A larger kern.ipc.sockbuf_waste_factor is checked at runtime. I have a patch with unit tests and documentation, tested in a FreeBSD 15.1 VM and on the router as above. I'd also propose adding --enable-ipfw to the FreeBSD CI job (qa/ci/freebsd.sh), since no CI job builds source-ipfw.c today. QUESTIONS FOR THE TEAM 1. IPFW is listed as unmaintained. Would you take this change? For my part, I'm willing to maintain the ipfw.buffer-size option and the divert socket setup that it interfaces with, and to handle bugs reported against them. I run Suricata in divert mode on my OPNsense router, and I can build and test IPFW changes in a FreeBSD VM. 2. Default: the patch keeps the kernel default unless the option is set, like af-packet and pcap. Would you rather have a default such as 1mb, which fits FreeBSD 14's default kern.ipc.maxsockbuf of 2 MiB? 3. Backport to 8.0.x: the change is opt-in, and the same diff applies to main-8.0.x. DISCLOSURE: I used Claude Fable 5.1 to support this investigation of this issue and generation of the code and tests to address, but I reviewed the generated code myself afterward and I ran the tests and measurements on my target hardware.