Setup¶
The proposed improvement can be demonstrated using Trex/Suricata setup. In my instance, those are 2 machines connected together (over switch but with the right MAC setup, it should not matter).
TREX.port0 -> switch -> Suricata.mlx0
I compared both AF-Packet + DPDK and tried to simulate heavy burst of e.g. 100k packets.
AF-Packet have more favorable settings, it has 8k descriptors, in comparison to DPDK's 256 or 1024 descriptors. I've noticed this later in the process but the improvement is still very well visible when comparing DPDK with/without the backlog.
Suricata setup with configured hugepages:
./autogen.sh && ./configure --enable-dpdk --enable-hwloc CFLAGS="-O3 -g" && make -j10
# Run afp-optimize-iface.sh before af-packet runs
sudo ./src/suricata -c suricata.yaml.afpdpdk.battle -l /tmp/ -vvvv -S ./test4.rules --dpdk
sudo ./src/suricata -c suricata.yaml.afpdpdk.battle -l /tmp/ -vvvv -S ./test4.rules --af-packet
Trex setup, from the installed location, e.g. /opt/trex/v3.06:
Window#1:
# allocate hugepages appropriately
echo 1024 | sudo tee /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
sudo ./t-rex-64 --cfg /tmp/trex_it1_1_cfg.yaml -i --stl -c 10
Window#2:
./trex-console -p 5601
# Then in trex interactive console shell
start -f stl/suri_microburst_udp.py -m 1 -d 10 -p 0 # single UDP flow, with heavy burst and a cool-off period afterwards.
Results¶
AFP with the maximum number of descriptors (8192) receives all replay packets. With 1024, it loses packets ~20%.
Initially I thought it is because of the ring-size setting (AFP Suricata.yaml option) but better AFP results are likely caused thanks to higher amount of RX descriptors.
DPDK with only 256/1024 descriptors and no backlog loses packets (~20% drop rate). With the backlog of 131k elements, 1024 descriptors receives all packets while 256 descriptors only lose ~0.5% packets.
Notes¶
- Mind that the results can be severely different on different machines but though manipulation of the Python scapy script, you could potentially reach similar results.
- This is a reproducer that works with lower descriptor counts -- it would be better if there would be one that works on 32k descriptors (maximums of ConnectX card) but I believe it demonstrates the case.
- Suricata uses 10 cores but one is sufficient as it is a single flows and only lands on one worker.
Trex machine: Intel(R) Xeon(R) Silver 4210
Suricata machine CPU: Intel(R) Xeon(R) CPU D-1557