Suricata

I agree TSO could be interesting to keep. What is your test ?

Test is iperf3:

Server:
iperf3 --server --port 20001

Client (running Suricata):
iperf3 -c <server-IPv4> -p 20001 -f m -i 1 -t 6 -O 2 -P 16

Result (truncated):
[ ID] Interval Transfer Bandwidth Retr
[SUM] 0.00-6.00 sec 6.55 GBytes 9381 Mbits/sec 6118 sender

In your test, you are testing the local stack not Suricata. In most cases, suricata is handling a copy of the traffic on an interface and not acting as a server.

You already can disable the offloading done by suricata by setting the folowing variables:

  capture:
    # disable NIC offloading. It's restored when Suricata exists.
    # Enabled by default
    disable-offloading: false

Eric Leblond wrote:

In your test, you are testing the local stack not Suricata. In most cases, suricata is handling a copy of the traffic on an interface and not acting as a server.

Not sure I follow. Suricata was running in the background:

suricata -D -c /etc/suricata/suricata.yaml -i eth0 --pidfile /var/log/suricata/suricata.pid

You already can disable the offloading done by suricata by setting the folowing variables:
[...]

Yes, I had already verified that, but note that:

1. It logs those warnings I mentioned in the original description of this issue
2. It leaves all offloads enabled, include GRO, LRO, GSO, SG, etc. not just TSO

What I'm proposing is specifically only leaving TSO enabled.

I'm not sure why I added TSO, or if it is necessary. I guess my thinking was that when capturing egress traffic it was relevant. So the question comes down to: is TSO handled before or after Suricata would capture egress traffic.

Victor Julien wrote:

I'm not sure why I added TSO, or if it is necessary. I guess my thinking was that when capturing egress traffic it was relevant. So the question comes down to: is TSO handled before or after Suricata would capture egress traffic.

So if I understand your question correctly, the actual TCP Segmentation is done after the protocol packet handlers are called, where pcap/AF_PACKET is hooked up in the Linux networking stack. So dev_queue_xmit (and helpers) first call the packet_rcv AF_PACKET protocol hook, before calling the NIC driver's ndo_start_xmit method.

For a TSO capable NIC, the NIC driver's ndo_start_xmit method will use the device-specific interface to set up TSO in hardware, and the NIC will DMA the un-segmented large TCP packet and do the TCP segmentation based on the MSS in hardware.

Please let me know if you agree with the proposed patches, or if I should make any changes.

I'm contemplating also leaving GSO enabled, because even though it's segmentation "offload" done in software, it was designed for better performance as described here: https://wiki.linuxfoundation.org/networking/gso. This will of course need to be tested, like I've already done for the other 2 patches to keep TSO and SG enabled.

I'm a bit confused about the goal here. If you want to keep offloads enabled, see https://redmine.openinfosecfoundation.org/issues/2218#note-4

The offloads give fake packets, and for accuracy we need the real packets. If that comes at a perf cost, so be it. Ppl that know what they are doing can disable the behavior.

I tested with TSO enabled, and it gives packets that are far bigger than the MTU. I guess AF_PACKET captures the larger packets that are sent to the NIC, that the NIC sends as real proper packets. So TSO needs to remain disabled so that capture of egress traffic works as expected. Not sure about sg. Disabling of gso is also needed on the receive side.

I'm not sure what "fake packets" v/s "real packets" means. The packets that suricata will see over the AF_PACKET interface will be exactly the ones that the NIC driver will be getting for transmit, as far as transmit side offloads such as TSO are concerned.

After that, the NIC driver will set up the NIC hardware to do the offload "on-the-fly" as it DMAs the packets from system memory and before it puts them on the wire. But system memory will never see the final TSO-segmented packets that will be put on the wire.

For that matter, this is no different than say a router in the middle doing IP fragmentation/reassembly; it's a piece of equipment in the end-to-end datapath that is free to further munge the packets in flight.

But from suricata's POV, it is seeing exactly the same packet that is being handed off to the NIC hardware by the NIC driver.

My issue with using the "disable-offloading: false" setting is the warnings suricata logs when you do that. I understand that this would be an issue with receive offloads such as GRO/LRO, where the NIC will indicate to it's driver that the checksum is "correct" without actually fixing the checksum in the packet delivered to system memory, and that information is lost across the AF_PACKET interface making the checksum appear invalid to suricata.

But for transmit side offloads, I'm yet to hear of a specific issue it can cause to suricata's abilities to analyze packets.

BTW even for receive checksum offloads, you can use the PACKET_AUXDATA socket option on AF_PACKET sockets, and look at TP_STATUS_CSUM* in aux.tp_status for what the NIC hardware determined w.r.t. the checksum. And looks like suricata's AFPReadFromRing function is doing this.

The point is that Suricata needs to see the packets as they are (& will be) on the wire.