Project

General

Profile

Actions

Bug #5771

open

xdp: Flows with nested VLANs are not bypassed by XDP filter

Added by Lukas Sismis almost 3 years ago. Updated 17 days ago.

Status:
New
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

Even though the eBPF (XDP) and Suricata structures are ready to handle nested VLANs (VLAN in VLAN) after my testing all packets were passed to Suricata.

I've found this when trying out TLS bypass - bypass TLS flow after TLS handshake. I am attaching a single TLS stream where after adding a VLAN all packets are forwarded to Suricata even though they should be bypassed after the handshake (after ~23 packets).


Files

shmu-tls-vlan-stream.pcap (439 KB) shmu-tls-vlan-stream.pcap Lukas Sismis, 01/02/2023 09:10 AM
shmu-tls-oneflow.pcap (436 KB) shmu-tls-oneflow.pcap Lukas Sismis, 09/22/2025 07:04 PM
suricata.xdp.yaml (91.1 KB) suricata.xdp.yaml Lukas Sismis, 09/22/2025 07:04 PM
Actions #1

Updated by Lukas Sismis almost 3 years ago

  • Subject changed from xdp: Flows with nested VLANs are not bypassed] to xdp: Flows with nested VLANs are not bypassed by XDP filter
Actions #2

Updated by Lukas Sismis almost 3 years ago

  • Description updated (diff)
Actions #3

Updated by Philippe Antoine about 1 month ago

Is this still the case in 8 ?

Actions #4

Updated by Philippe Antoine about 1 month ago

  • Status changed from New to Feedback

Seems to be working for me :

Using a rule alert tls any any -> any any (sid: 43; bypass;). and your pcap I get no tls event and
```
"flow_bypassed": {
"local_pkts": 0,
"local_bytes": 0,
"local_capture_pkts": 65,
"local_capture_bytes": 27835,
```

and without using the rule, I get the tls event and zeroes in stats bypass

Updated by Lukas Sismis 17 days ago

Yes, it is present in 9.0.x main branch.

Test topology - two ports connected back-to-back (a loopback between 2 ports on the same NIC)

To reproduce:

  • Compiled Suricata with eBPF support:
    CC=clang ./configure --prefix=/usr/ --sysconfdir=/etc/ --localstatedir=/var/ --enable-ebpf --enable-ebpf-build
    
  • Ran Suricata:
    sudo ./src/suricata --af-packet -c suricata.xdp.yaml -S /dev/null -l /tmp/ -vvvv
    

Attaching the used configuration file (tried on Suricata 9.0.x main).
There is no need to create an extra TLS rule with bypass keyword, the point is of the test is to let Suricata detect encrypted part and then bypass the flow (encryption-handling).

One pcap has no VLAN and is correctly bypassed after TLS handshake in the XDP program. The other PCAP (the originally attached) with VLAN is forwarded to Suricata where the traffic is locally bypassed.

no VLAN

TX send:

$ sudo tcpreplay -i ens2f0np0 /tmp/shmu-tls-oneflow.pcap 
Actual: 599 packets (437107 bytes) sent in 1.89 seconds
Rated: 230638.7 Bps, 1.84 Mbps, 316.06 pps
Flows: 2 flows, 1.05 fps, 599 unique flow packets, 0 unique non-flow packets
Statistics for network device: ens2f0np0
    Successful packets:        599
    Failed packets:            0
    Truncated packets:         0
    Retried packets (ENOBUFS): 0
    Retried packets (EAGAIN):  0

RX logs:

Info: ioctl: ens2f1np1: MTU 9000 [GetIfaceMTU:util-ioctl.c:101]
Perf: af-packet: ens2f1np1: rx ring params: block_size=131072 block_nr=147 frame_size=9120 frame_nr=2058 (mem: 19267584) [AFPComputeRingParamsV3:source-af-packet.c:1699]
Notice: threads: Threads created -> W: 1 FM: 1 FR: 1   Engine started. [TmThreadWaitOnThreadRunning:tm-threads.c:1982]
^CNotice: suricata: Signal Received.  Stopping engine. [SuricataMainLoop:suricata.c:2931]
Info: suricata: time elapsed 9.041s [SCPrintElapsedTime:suricata.c:1228]
Perf: flow-manager: 0 flows processed [FlowRecycler:flow-manager.c:1177]
Perf: af-packet: ens2f1np1: (W#01-ens2f1np1) kernel: Packets 22, dropped 0 [ReceiveAFPThreadExitStats:source-af-packet.c:2699]
Perf: detect: threshold thread cache stats: cnt:0 notinit:0 nosupport:0 miss_expired:0 miss:0 hit:0, housekeeping: checks:0, expired:0 [DumpCacheStats:detect-engine-threshold.c:271]
Info: counters: Alerts: 0 [StatsLogSummary:counters.c:869]
Perf: ippair: ippair memory usage: 398144 bytes, maximum: 16777216 [IPPairPrintStats:ippair.c:285]
Perf: host: host memory usage: 382144 bytes, maximum: 33554432 [HostPrintStats:host.c:291]
Notice: device: ens2f1np1: packets: 22, drops: 0 (0.00%), invalid chksum: 0 [LiveDeviceListClean:util-device.c:351]

stats:

capture.kernel_packets                                       | Total                     | 22
capture.afpacket.polls                                       | Total                     | 90
capture.afpacket.poll_timeout                                | Total                     | 83
capture.afpacket.poll_data                                   | Total                     | 6
decoder.pkts                                                 | Total                     | 22
decoder.bytes                                                | Total                     | 7757
decoder.ipv4                                                 | Total                     | 22
decoder.ethernet                                             | Total                     | 22
decoder.tcp                                                  | Total                     | 22
tcp.syn                                                      | Total                     | 1
tcp.synack                                                   | Total                     | 1
decoder.avg_pkt_size                                         | Total                     | 352
decoder.max_pkt_size                                         | Total                     | 1466
flow.total                                                   | Total                     | 1
flow.tcp                                                     | Total                     | 1
flow.wrk.spare_sync_avg                                      | Total                     | 100
flow.wrk.spare_sync                                          | Total                     | 1
flow_bypassed.local_capture_pkts                             | Total                     | 3
flow_bypassed.local_capture_bytes                            | Total                     | 745

Because the XDP bypass filter works, you don't see the remaining packets.

VLAN present

TX:

sudo tcpreplay -i ens2f0np0 /tmp/shmu-tls-vlan-stream.pcap 
Actual: 599 packets (439503 bytes) sent in 1.89 seconds
Rated: 231899.1 Bps, 1.85 Mbps, 316.05 pps
Flows: 2 flows, 1.05 fps, 599 unique flow packets, 0 unique non-flow packets
Statistics for network device: ens2f0np0
    Successful packets:        599
    Failed packets:            0
    Truncated packets:         0
    Retried packets (ENOBUFS): 0
    Retried packets (EAGAIN):  0

RX logs:

Info: ioctl: ens2f1np1: MTU 9000 [GetIfaceMTU:util-ioctl.c:101]
Perf: af-packet: ens2f1np1: rx ring params: block_size=131072 block_nr=147 frame_size=9120 frame_nr=2058 (mem: 19267584) [AFPComputeRingParamsV3:source-af-packet.c:1699]
Notice: threads: Threads created -> W: 1 FM: 1 FR: 1   Engine started. [TmThreadWaitOnThreadRunning:tm-threads.c:1982]
^CNotice: suricata: Signal Received.  Stopping engine. [SuricataMainLoop:suricata.c:2931]
Info: suricata: time elapsed 13.018s [SCPrintElapsedTime:suricata.c:1228]
Perf: flow-manager: 0 flows processed [FlowRecycler:flow-manager.c:1177]
Perf: af-packet: ens2f1np1: (W#01-ens2f1np1) kernel: Packets 599, dropped 0 [ReceiveAFPThreadExitStats:source-af-packet.c:2699]
Perf: detect: threshold thread cache stats: cnt:0 notinit:0 nosupport:0 miss_expired:0 miss:0 hit:0, housekeeping: checks:0, expired:0 [DumpCacheStats:detect-engine-threshold.c:271]
Info: counters: Alerts: 0 [StatsLogSummary:counters.c:869]
Perf: ippair: ippair memory usage: 398144 bytes, maximum: 16777216 [IPPairPrintStats:ippair.c:285]
Perf: host: host memory usage: 382144 bytes, maximum: 33554432 [HostPrintStats:host.c:291]
Notice: device: ens2f1np1: packets: 599, drops: 0 (0.00%), invalid chksum: 0 [LiveDeviceListClean:util-device.c:351]

RX stats:

capture.kernel_packets                                       | Total                     | 599
capture.afpacket.polls                                       | Total                     | 145
capture.afpacket.poll_timeout                                | Total                     | 113
capture.afpacket.poll_data                                   | Total                     | 31
decoder.pkts                                                 | Total                     | 599
decoder.bytes                                                | Total                     | 437113
decoder.ipv4                                                 | Total                     | 599
decoder.ethernet                                             | Total                     | 599
decoder.tcp                                                  | Total                     | 599
tcp.syn                                                      | Total                     | 1
tcp.synack                                                   | Total                     | 1
tcp.rst                                                      | Total                     | 3
decoder.vlan                                                 | Total                     | 599
decoder.avg_pkt_size                                         | Total                     | 729
decoder.max_pkt_size                                         | Total                     | 1466
flow.total                                                   | Total                     | 1
flow.tcp                                                     | Total                     | 1
flow.wrk.spare_sync_avg                                      | Total                     | 100
flow.wrk.spare_sync                                          | Total                     | 1
flow_bypassed.local_capture_pkts                             | Total                     | 580
flow_bypassed.local_capture_bytes                            | Total                     | 432421

In this case you see that all packets are passed to SW Suricata but they are still rightfully locally bypassed.

Actions #6

Updated by Jason Ish 17 days ago

I'm not the familar with XDP, but is this a Suricata issue? Or an issue with the ebpf/xdp program loaded into the kernel?

Actions #7

Updated by Lukas Sismis 17 days ago

I assume the issue is in the XDP program that does bypass and is part of Suricata codebase (in the ebpf/ folder).
When referring to XDP, I am talking about AF_PACKET + XDP (eBPF program) and not AF_XDP.

Actions

Also available in: Atom PDF