Bug #5771
openxdp: Flows with nested VLANs are not bypassed by XDP filter
Added by Lukas Sismis almost 3 years ago. Updated 17 days ago.
Description
Even though the eBPF (XDP) and Suricata structures are ready to handle nested VLANs (VLAN in VLAN) after my testing all packets were passed to Suricata.
I've found this when trying out TLS bypass - bypass TLS flow after TLS handshake. I am attaching a single TLS stream where after adding a VLAN all packets are forwarded to Suricata even though they should be bypassed after the handshake (after ~23 packets).
Files
| shmu-tls-vlan-stream.pcap (439 KB) shmu-tls-vlan-stream.pcap | Lukas Sismis, 01/02/2023 09:10 AM | ||
| shmu-tls-oneflow.pcap (436 KB) shmu-tls-oneflow.pcap | Lukas Sismis, 09/22/2025 07:04 PM | ||
| suricata.xdp.yaml (91.1 KB) suricata.xdp.yaml | Lukas Sismis, 09/22/2025 07:04 PM |
Updated by Lukas Sismis almost 3 years ago
- Subject changed from xdp: Flows with nested VLANs are not bypassed] to xdp: Flows with nested VLANs are not bypassed by XDP filter
Updated by Philippe Antoine about 1 month ago
- Status changed from New to Feedback
Seems to be working for me :
Using a rule alert tls any any -> any any (sid: 43; bypass;). and your pcap I get no tls event and
```
"flow_bypassed": {
"local_pkts": 0,
"local_bytes": 0,
"local_capture_pkts": 65,
"local_capture_bytes": 27835,
```
and without using the rule, I get the tls event and zeroes in stats bypass
Updated by Lukas Sismis 17 days ago
- File shmu-tls-oneflow.pcap shmu-tls-oneflow.pcap added
- File suricata.xdp.yaml suricata.xdp.yaml added
- Status changed from Feedback to New
- Affected Versions 7.0.12, 8.0.1, 9.0.0-beta1 added
Yes, it is present in 9.0.x main branch.
Test topology - two ports connected back-to-back (a loopback between 2 ports on the same NIC)
To reproduce:
- Installed libbpf - https://docs.suricata.io/en/latest/capture-hardware/ebpf-xdp.html#libbpf
- Compiled Suricata with eBPF support:
CC=clang ./configure --prefix=/usr/ --sysconfdir=/etc/ --localstatedir=/var/ --enable-ebpf --enable-ebpf-build
- Set up XDP bypass - https://docs.suricata.io/en/latest/capture-hardware/ebpf-xdp.html#setup-xdp-bypass
- Ran Suricata:
sudo ./src/suricata --af-packet -c suricata.xdp.yaml -S /dev/null -l /tmp/ -vvvv
Attaching the used configuration file (tried on Suricata 9.0.x main).
There is no need to create an extra TLS rule with bypass keyword, the point is of the test is to let Suricata detect encrypted part and then bypass the flow (encryption-handling).
One pcap has no VLAN and is correctly bypassed after TLS handshake in the XDP program. The other PCAP (the originally attached) with VLAN is forwarded to Suricata where the traffic is locally bypassed.
no VLAN¶
TX send:
$ sudo tcpreplay -i ens2f0np0 /tmp/shmu-tls-oneflow.pcap
Actual: 599 packets (437107 bytes) sent in 1.89 seconds
Rated: 230638.7 Bps, 1.84 Mbps, 316.06 pps
Flows: 2 flows, 1.05 fps, 599 unique flow packets, 0 unique non-flow packets
Statistics for network device: ens2f0np0
Successful packets: 599
Failed packets: 0
Truncated packets: 0
Retried packets (ENOBUFS): 0
Retried packets (EAGAIN): 0
RX logs:
Info: ioctl: ens2f1np1: MTU 9000 [GetIfaceMTU:util-ioctl.c:101] Perf: af-packet: ens2f1np1: rx ring params: block_size=131072 block_nr=147 frame_size=9120 frame_nr=2058 (mem: 19267584) [AFPComputeRingParamsV3:source-af-packet.c:1699] Notice: threads: Threads created -> W: 1 FM: 1 FR: 1 Engine started. [TmThreadWaitOnThreadRunning:tm-threads.c:1982] ^CNotice: suricata: Signal Received. Stopping engine. [SuricataMainLoop:suricata.c:2931] Info: suricata: time elapsed 9.041s [SCPrintElapsedTime:suricata.c:1228] Perf: flow-manager: 0 flows processed [FlowRecycler:flow-manager.c:1177] Perf: af-packet: ens2f1np1: (W#01-ens2f1np1) kernel: Packets 22, dropped 0 [ReceiveAFPThreadExitStats:source-af-packet.c:2699] Perf: detect: threshold thread cache stats: cnt:0 notinit:0 nosupport:0 miss_expired:0 miss:0 hit:0, housekeeping: checks:0, expired:0 [DumpCacheStats:detect-engine-threshold.c:271] Info: counters: Alerts: 0 [StatsLogSummary:counters.c:869] Perf: ippair: ippair memory usage: 398144 bytes, maximum: 16777216 [IPPairPrintStats:ippair.c:285] Perf: host: host memory usage: 382144 bytes, maximum: 33554432 [HostPrintStats:host.c:291] Notice: device: ens2f1np1: packets: 22, drops: 0 (0.00%), invalid chksum: 0 [LiveDeviceListClean:util-device.c:351]
stats:
capture.kernel_packets | Total | 22 capture.afpacket.polls | Total | 90 capture.afpacket.poll_timeout | Total | 83 capture.afpacket.poll_data | Total | 6 decoder.pkts | Total | 22 decoder.bytes | Total | 7757 decoder.ipv4 | Total | 22 decoder.ethernet | Total | 22 decoder.tcp | Total | 22 tcp.syn | Total | 1 tcp.synack | Total | 1 decoder.avg_pkt_size | Total | 352 decoder.max_pkt_size | Total | 1466 flow.total | Total | 1 flow.tcp | Total | 1 flow.wrk.spare_sync_avg | Total | 100 flow.wrk.spare_sync | Total | 1 flow_bypassed.local_capture_pkts | Total | 3 flow_bypassed.local_capture_bytes | Total | 745
Because the XDP bypass filter works, you don't see the remaining packets.
VLAN present¶
TX:
sudo tcpreplay -i ens2f0np0 /tmp/shmu-tls-vlan-stream.pcap
Actual: 599 packets (439503 bytes) sent in 1.89 seconds
Rated: 231899.1 Bps, 1.85 Mbps, 316.05 pps
Flows: 2 flows, 1.05 fps, 599 unique flow packets, 0 unique non-flow packets
Statistics for network device: ens2f0np0
Successful packets: 599
Failed packets: 0
Truncated packets: 0
Retried packets (ENOBUFS): 0
Retried packets (EAGAIN): 0
RX logs:
Info: ioctl: ens2f1np1: MTU 9000 [GetIfaceMTU:util-ioctl.c:101] Perf: af-packet: ens2f1np1: rx ring params: block_size=131072 block_nr=147 frame_size=9120 frame_nr=2058 (mem: 19267584) [AFPComputeRingParamsV3:source-af-packet.c:1699] Notice: threads: Threads created -> W: 1 FM: 1 FR: 1 Engine started. [TmThreadWaitOnThreadRunning:tm-threads.c:1982] ^CNotice: suricata: Signal Received. Stopping engine. [SuricataMainLoop:suricata.c:2931] Info: suricata: time elapsed 13.018s [SCPrintElapsedTime:suricata.c:1228] Perf: flow-manager: 0 flows processed [FlowRecycler:flow-manager.c:1177] Perf: af-packet: ens2f1np1: (W#01-ens2f1np1) kernel: Packets 599, dropped 0 [ReceiveAFPThreadExitStats:source-af-packet.c:2699] Perf: detect: threshold thread cache stats: cnt:0 notinit:0 nosupport:0 miss_expired:0 miss:0 hit:0, housekeeping: checks:0, expired:0 [DumpCacheStats:detect-engine-threshold.c:271] Info: counters: Alerts: 0 [StatsLogSummary:counters.c:869] Perf: ippair: ippair memory usage: 398144 bytes, maximum: 16777216 [IPPairPrintStats:ippair.c:285] Perf: host: host memory usage: 382144 bytes, maximum: 33554432 [HostPrintStats:host.c:291] Notice: device: ens2f1np1: packets: 599, drops: 0 (0.00%), invalid chksum: 0 [LiveDeviceListClean:util-device.c:351]
RX stats:
capture.kernel_packets | Total | 599 capture.afpacket.polls | Total | 145 capture.afpacket.poll_timeout | Total | 113 capture.afpacket.poll_data | Total | 31 decoder.pkts | Total | 599 decoder.bytes | Total | 437113 decoder.ipv4 | Total | 599 decoder.ethernet | Total | 599 decoder.tcp | Total | 599 tcp.syn | Total | 1 tcp.synack | Total | 1 tcp.rst | Total | 3 decoder.vlan | Total | 599 decoder.avg_pkt_size | Total | 729 decoder.max_pkt_size | Total | 1466 flow.total | Total | 1 flow.tcp | Total | 1 flow.wrk.spare_sync_avg | Total | 100 flow.wrk.spare_sync | Total | 1 flow_bypassed.local_capture_pkts | Total | 580 flow_bypassed.local_capture_bytes | Total | 432421
In this case you see that all packets are passed to SW Suricata but they are still rightfully locally bypassed.
Updated by Lukas Sismis 17 days ago
I assume the issue is in the XDP program that does bypass and is part of Suricata codebase (in the ebpf/ folder).
When referring to XDP, I am talking about AF_PACKET + XDP (eBPF program) and not AF_XDP.