Actions
Bug #6731
openeBPF XDP program is not attached when pinned-maps is true
Affected Versions:
Effort:
Difficulty:
Label:
Description
when start Suricata with xdp_filter and AF_PACKET IPS mode with pinned-maps set to true as below config:
af-packet:
- interface: eno1
threads: auto
cluster-id: 99
cluster-type: cluster_qm #for IPS
# cluster-type: cluster_flow #for IDS
defrag: yes
# copy-mode: tap
copy-mode: ips
copy-iface: eno2
xdp-mode: driver
pinned-maps: true
# pinned-maps-name: flow_table_v4
xdp-filter-file: /usr/libexec/suricata/ebpf/xdp_filter.bpf
# bypass: yes
use-mmap: yes
ring-size: 200000
buffer-size: 64535
- interface: eno2
threads: auto
cluster-id: 100
cluster-type: cluster_qm
defrag: yes
copy-mode: ips
copy-iface: eno1
xdp-mode: driver
pinned-maps: true
# pinned-maps-name: flow_table_v4
xdp-filter-file: /usr/libexec/suricata/ebpf/xdp_filter.bpf
# bypass: yes
use-mmap: yes
ring-size: 200000
buffer-size: 64535
root@r210:~# suricata -c /etc/suricata/suricata-ips.yaml --af-packet -vvv
Info: af-packet: eno1: AF_PACKET IPS mode activated eno1->eno2 [ParseAFPConfig:runmode-af-packet.c:353] Config: af-packet: eno1: using queue based cluster mode for AF_PACKET [ParseAFPConfig:runmode-af-packet.c:410] Config: af-packet: eno1: using pinned maps [ParseAFPConfig:runmode-af-packet.c:454] Config: ebpf: Pinning: 8 to flow_table_v4 [EBPFLoadFile:util-ebpf.c:430] Config: ebpf: Pinning: 9 to flow_table_v6 [EBPFLoadFile:util-ebpf.c:430] Config: ebpf: Pinning: 10 to cpu_map [EBPFLoadFile:util-ebpf.c:430] Config: ebpf: Pinning: 11 to cpus_available [EBPFLoadFile:util-ebpf.c:430] Config: ebpf: Pinning: 12 to cpus_count [EBPFLoadFile:util-ebpf.c:430] Config: ebpf: Pinning: 13 to tx_peer [EBPFLoadFile:util-ebpf.c:430] Config: ebpf: Pinning: 14 to tx_peer_int [EBPFLoadFile:util-ebpf.c:430] Config: ebpf: Pinning: 15 to .rodata.str1.1 [EBPFLoadFile:util-ebpf.c:430] Warning: ebpf: Can not pin: Operation not permitted [EBPFLoadFile:util-ebpf.c:436] Info: ebpf: Successfully loaded eBPF file '/usr/libexec/suricata/ebpf/xdp_filter.bpf' on 'eno1' [EBPFLoadFile:util-ebpf.c:461] Info: ioctl: eno1: RX RSS queues: 8 [GetIfaceRSSQueuesNum:util-ioctl.c:734] Perf: af-packet: eno1: cluster_qm: 8 RSS queues, using 8 threads [ParseAFPConfig:runmode-af-packet.c:727] Info: runmodes: eno1: creating 1 thread [RunModeSetLiveCaptureWorkersForDevice:util-runmodes.c:254] Info: af-packet: eno2: AF_PACKET IPS mode activated eno2->eno1 [ParseAFPConfig:runmode-af-packet.c:353] Config: af-packet: eno2: using queue based cluster mode for AF_PACKET [ParseAFPConfig:runmode-af-packet.c:410] Config: af-packet: eno2: using pinned maps [ParseAFPConfig:runmode-af-packet.c:454] Config: ebpf: Pinning: 18 to flow_table_v4 [EBPFLoadFile:util-ebpf.c:430] Config: ebpf: Pinning: 19 to flow_table_v6 [EBPFLoadFile:util-ebpf.c:430] Config: ebpf: Pinning: 20 to cpu_map [EBPFLoadFile:util-ebpf.c:430] Config: ebpf: Pinning: 21 to cpus_available [EBPFLoadFile:util-ebpf.c:430] Config: ebpf: Pinning: 22 to cpus_count [EBPFLoadFile:util-ebpf.c:430] Config: ebpf: Pinning: 23 to tx_peer [EBPFLoadFile:util-ebpf.c:430] Config: ebpf: Pinning: 24 to tx_peer_int [EBPFLoadFile:util-ebpf.c:430] Config: ebpf: Pinning: 25 to .rodata.str1.1 [EBPFLoadFile:util-ebpf.c:430] Warning: ebpf: Can not pin: Operation not permitted [EBPFLoadFile:util-ebpf.c:436] Info: ebpf: Successfully loaded eBPF file '/usr/libexec/suricata/ebpf/xdp_filter.bpf' on 'eno2' [EBPFLoadFile:util-ebpf.c:461] Info: ioctl: eno2: RX RSS queues: 8 [GetIfaceRSSQueuesNum:util-ioctl.c:734] Perf: af-packet: eno2: cluster_qm: 8 RSS queues, using 8 threads [ParseAFPConfig:runmode-af-packet.c:727] Info: runmodes: eno2: creating 1 thread [RunModeSetLiveCaptureWorkersForDevice:util-runmodes.c:254] Config: flow-manager: using 1 flow manager threads [FlowManagerThreadSpawn:flow-manager.c:959] Config: flow-manager: using 1 flow recycler threads [FlowRecyclerThreadSpawn:flow-manager.c:1162] Info: unix-manager: unix socket '/var/run/suricata/suricata-command.socket' [UnixNew:unix-manager.c:136] Perf: af-packet: eno1: setting socket buffer to 64535 [AFPCreateSocket:source-af-packet.c:1950] Perf: af-packet: eno1: rx ring: block_size=32768 block_nr=10001 frame_size=1600 frame_nr=200020 [AFPComputeRingParams:source-af-packet.c:1600] Perf: af-packet: eno2: setting socket buffer to 64535 [AFPCreateSocket:source-af-packet.c:1950] Perf: af-packet: eno2: rx ring: block_size=32768 block_nr=10001 frame_size=1600 frame_nr=200020 [AFPComputeRingParams:source-af-packet.c:1600] Notice: threads: Threads created -> W: 2 FM: 1 FR: 1 Engine started. [TmThreadWaitOnThreadRunning:tm-threads.c:1891]
open another ssh session to run xdp-loader status to show the xdp program is attached ok
root@r210:~# xdp-loader status CURRENT XDP PROGRAM STATUS: Interface Prio Program name Mode ID Tag Chain actions -------------------------------------------------------------------------------------- lo <No XDP program loaded!> enp2s0f0 <No XDP program loaded!> enp2s0f1 <No XDP program loaded!> eno1 xdp_hashfilter native 319 f1787f7950a1278f eno2 xdp_hashfilter native 322 f1787f7950a1278f br0 <No XDP program loaded!> wg0 <No XDP program loaded!> virbr0 <No XDP program loaded!>
then run xdp-loader to detach the program manually
root@r210:~# xdp-loader unload eno1 -a root@r210:~# xdp-loader status CURRENT XDP PROGRAM STATUS: Interface Prio Program name Mode ID Tag Chain actions -------------------------------------------------------------------------------------- lo <No XDP program loaded!> enp2s0f0 <No XDP program loaded!> enp2s0f1 <No XDP program loaded!> eno1 <No XDP program loaded!> eno2 xdp_hashfilter native 322 f1787f7950a1278f br0 <No XDP program loaded!> wg0 <No XDP program loaded!> virbr0 <No XDP program loaded!>go back to the ssh session window where Suricata is started, control + c to stop Suricata
^CNotice: suricata: Signal Received. Stopping engine. [SuricataMainLoop:suricata.c:2811] Info: suricata: time elapsed 460.567s [SCPrintElapsedTime:suricata.c:1167] Perf: flow-manager: 29 flows processed [FlowRecycler:flow-manager.c:1134] Perf: af-packet: eno1: (W#01-eno1) kernel: Packets 28, dropped 0 [ReceiveAFPThreadExitStats:source-af-packet.c:2631] Perf: af-packet: eno2: (W#01-eno2) kernel: Packets 40, dropped 0 [ReceiveAFPThreadExitStats:source-af-packet.c:2631] Info: counters: Alerts: 0 [StatsLogSummary:counters.c:888] Perf: ippair: ippair memory usage: 406144 bytes, maximum: 16777216 [IPPairPrintStats:ippair.c:295] Perf: host: host memory usage: 390144 bytes, maximum: 33554432 [HostPrintStats:host.c:298] Perf: app-layer-htp: htp memory 0 (0) [AppLayerHtpPrintStats:app-layer-htp.c:3036] Notice: device: eno1: packets: 28, drops: 0 (0.00%), invalid chksum: 0 [LiveDeviceListClean:util-device.c:331] Notice: device: eno2: packets: 40, drops: 0 (0.00%), invalid chksum: 0 [LiveDeviceListClean:util-device.c:331]
start Suricata again:
Info: af-packet: eno1: AF_PACKET IPS mode activated eno1->eno2 [ParseAFPConfig:runmode-af-packet.c:353] Config: af-packet: eno1: using queue based cluster mode for AF_PACKET [ParseAFPConfig:runmode-af-packet.c:410] Config: af-packet: eno1: using pinned maps [ParseAFPConfig:runmode-af-packet.c:454] Info: ebpf: Loaded pinned maps, will use already loaded eBPF filter [EBPFLoadFile:util-ebpf.c:317] Info: af-packet: eno1: loaded pinned maps from sysfs [ParseAFPConfig:runmode-af-packet.c:596] Info: ioctl: eno1: RX RSS queues: 8 [GetIfaceRSSQueuesNum:util-ioctl.c:734] Perf: af-packet: eno1: cluster_qm: 8 RSS queues, using 8 threads [ParseAFPConfig:runmode-af-packet.c:727] Info: runmodes: eno1: creating 1 thread [RunModeSetLiveCaptureWorkersForDevice:util-runmodes.c:254] Info: af-packet: eno2: AF_PACKET IPS mode activated eno2->eno1 [ParseAFPConfig:runmode-af-packet.c:353] Config: af-packet: eno2: using queue based cluster mode for AF_PACKET [ParseAFPConfig:runmode-af-packet.c:410] Config: af-packet: eno2: using pinned maps [ParseAFPConfig:runmode-af-packet.c:454] Info: ebpf: Loaded pinned maps, will use already loaded eBPF filter [EBPFLoadFile:util-ebpf.c:317] Info: af-packet: eno2: loaded pinned maps from sysfs [ParseAFPConfig:runmode-af-packet.c:596] Info: ioctl: eno2: RX RSS queues: 8 [GetIfaceRSSQueuesNum:util-ioctl.c:734] Perf: af-packet: eno2: cluster_qm: 8 RSS queues, using 8 threads [ParseAFPConfig:runmode-af-packet.c:727] Info: runmodes: eno2: creating 1 thread [RunModeSetLiveCaptureWorkersForDevice:util-runmodes.c:254] Config: flow-manager: using 1 flow manager threads [FlowManagerThreadSpawn:flow-manager.c:959] Config: flow-manager: using 1 flow recycler threads [FlowRecyclerThreadSpawn:flow-manager.c:1162] Info: unix-manager: unix socket '/var/run/suricata/suricata-command.socket' [UnixNew:unix-manager.c:136] Perf: af-packet: eno1: setting socket buffer to 64535 [AFPCreateSocket:source-af-packet.c:1950] Perf: af-packet: eno1: rx ring: block_size=32768 block_nr=10001 frame_size=1600 frame_nr=200020 [AFPComputeRingParams:source-af-packet.c:1600] Perf: af-packet: eno2: setting socket buffer to 64535 [AFPCreateSocket:source-af-packet.c:1950] Perf: af-packet: eno2: rx ring: block_size=32768 block_nr=10001 frame_size=1600 frame_nr=200020 [AFPComputeRingParams:source-af-packet.c:1600] Notice: threads: Threads created -> W: 2 FM: 1 FR: 1 Engine started. [TmThreadWaitOnThreadRunning:tm-threads.c:1891]
xdp-loader status shows xdp program is not attached to eno1 interface
root@r210:~# xdp-loader status CURRENT XDP PROGRAM STATUS: Interface Prio Program name Mode ID Tag Chain actions -------------------------------------------------------------------------------------- lo <No XDP program loaded!> enp2s0f0 <No XDP program loaded!> enp2s0f1 <No XDP program loaded!> eno1 <No XDP program loaded!> eno2 xdp_hashfilter native 322 f1787f7950a1278f br0 <No XDP program loaded!> wg0 <No XDP program loaded!> virbr0 <No XDP program loaded!>
the reason is EBPFLoadFile will not open/load/attach XDP program when pinned-maps is set to true
299 int EBPFLoadFile(const char *iface, const char *path, const char * section,
300 int *val, struct ebpf_timeout_config *config)
301 {
302 int err, pfd;
303 bool found = false;
304 struct bpf_object *bpfobj = NULL;
305 struct bpf_program *bpfprog = NULL;
306 struct bpf_map *map = NULL;
307
308 if (iface == NULL)
309 return -1;
310 LiveDevice *livedev = LiveGetDevice(iface);
311 if (livedev == NULL)
312 return -1;
313
314 if (config->flags & EBPF_XDP_CODE && config->flags & EBPF_PINNED_MAPS) { <====================
315 /* We try to get our flow table maps and if we have them we can simply return */
316 if (EBPFLoadPinnedMaps(livedev, config) == 0) {. <========================================
317 SCLogInfo("Loaded pinned maps, will use already loaded eBPF filter");
318 return 1;
319 }
320 }
I know this is corner case that users are not suppose to unload the XDP program manually with other xdp utility program, but for whatever unexpected situation, if the XDP program is detached, when suricata is started, suricata should check if program is attached when pinned-maps is set to true.
Updated by Victor Julien 3 months ago
@Vincent Li are you planning to work on this?
Actions