Project

General

Profile

Actions

Bug #6731

open

eBPF XDP program is not attached when pinned-maps is true

Added by Vincent Li 5 months ago. Updated 4 months ago.

Status:
New
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

when start Suricata with xdp_filter and AF_PACKET IPS mode with pinned-maps set to true as below config:

af-packet:
  - interface: eno1
    threads: auto
    cluster-id: 99
    cluster-type: cluster_qm #for IPS
#    cluster-type: cluster_flow #for IDS
    defrag: yes
#    copy-mode: tap 
    copy-mode: ips
    copy-iface: eno2
    xdp-mode: driver
    pinned-maps: true
#    pinned-maps-name: flow_table_v4
    xdp-filter-file:  /usr/libexec/suricata/ebpf/xdp_filter.bpf
#    bypass: yes
    use-mmap: yes
    ring-size: 200000
    buffer-size: 64535

  - interface: eno2
    threads: auto
    cluster-id: 100
    cluster-type: cluster_qm
    defrag: yes
    copy-mode: ips
    copy-iface: eno1
    xdp-mode: driver
    pinned-maps: true
#    pinned-maps-name: flow_table_v4
    xdp-filter-file:  /usr/libexec/suricata/ebpf/xdp_filter.bpf
#    bypass: yes
    use-mmap: yes
    ring-size: 200000
    buffer-size: 64535

root@r210:~# suricata -c /etc/suricata/suricata-ips.yaml --af-packet -vvv

Info: af-packet: eno1: AF_PACKET IPS mode activated eno1->eno2 [ParseAFPConfig:runmode-af-packet.c:353]
Config: af-packet: eno1: using queue based cluster mode for AF_PACKET [ParseAFPConfig:runmode-af-packet.c:410]
Config: af-packet: eno1: using pinned maps [ParseAFPConfig:runmode-af-packet.c:454]
Config: ebpf: Pinning: 8 to flow_table_v4 [EBPFLoadFile:util-ebpf.c:430]
Config: ebpf: Pinning: 9 to flow_table_v6 [EBPFLoadFile:util-ebpf.c:430]
Config: ebpf: Pinning: 10 to cpu_map [EBPFLoadFile:util-ebpf.c:430]
Config: ebpf: Pinning: 11 to cpus_available [EBPFLoadFile:util-ebpf.c:430]
Config: ebpf: Pinning: 12 to cpus_count [EBPFLoadFile:util-ebpf.c:430]
Config: ebpf: Pinning: 13 to tx_peer [EBPFLoadFile:util-ebpf.c:430]
Config: ebpf: Pinning: 14 to tx_peer_int [EBPFLoadFile:util-ebpf.c:430]
Config: ebpf: Pinning: 15 to .rodata.str1.1 [EBPFLoadFile:util-ebpf.c:430]
Warning: ebpf: Can not pin: Operation not permitted [EBPFLoadFile:util-ebpf.c:436]
Info: ebpf: Successfully loaded eBPF file '/usr/libexec/suricata/ebpf/xdp_filter.bpf' on 'eno1' [EBPFLoadFile:util-ebpf.c:461]
Info: ioctl: eno1: RX RSS queues: 8 [GetIfaceRSSQueuesNum:util-ioctl.c:734]
Perf: af-packet: eno1: cluster_qm: 8 RSS queues, using 8 threads [ParseAFPConfig:runmode-af-packet.c:727]
Info: runmodes: eno1: creating 1 thread [RunModeSetLiveCaptureWorkersForDevice:util-runmodes.c:254]
Info: af-packet: eno2: AF_PACKET IPS mode activated eno2->eno1 [ParseAFPConfig:runmode-af-packet.c:353]
Config: af-packet: eno2: using queue based cluster mode for AF_PACKET [ParseAFPConfig:runmode-af-packet.c:410]
Config: af-packet: eno2: using pinned maps [ParseAFPConfig:runmode-af-packet.c:454]
Config: ebpf: Pinning: 18 to flow_table_v4 [EBPFLoadFile:util-ebpf.c:430]
Config: ebpf: Pinning: 19 to flow_table_v6 [EBPFLoadFile:util-ebpf.c:430]
Config: ebpf: Pinning: 20 to cpu_map [EBPFLoadFile:util-ebpf.c:430]
Config: ebpf: Pinning: 21 to cpus_available [EBPFLoadFile:util-ebpf.c:430]
Config: ebpf: Pinning: 22 to cpus_count [EBPFLoadFile:util-ebpf.c:430]
Config: ebpf: Pinning: 23 to tx_peer [EBPFLoadFile:util-ebpf.c:430]
Config: ebpf: Pinning: 24 to tx_peer_int [EBPFLoadFile:util-ebpf.c:430]
Config: ebpf: Pinning: 25 to .rodata.str1.1 [EBPFLoadFile:util-ebpf.c:430]
Warning: ebpf: Can not pin: Operation not permitted [EBPFLoadFile:util-ebpf.c:436]
Info: ebpf: Successfully loaded eBPF file '/usr/libexec/suricata/ebpf/xdp_filter.bpf' on 'eno2' [EBPFLoadFile:util-ebpf.c:461]
Info: ioctl: eno2: RX RSS queues: 8 [GetIfaceRSSQueuesNum:util-ioctl.c:734]
Perf: af-packet: eno2: cluster_qm: 8 RSS queues, using 8 threads [ParseAFPConfig:runmode-af-packet.c:727]
Info: runmodes: eno2: creating 1 thread [RunModeSetLiveCaptureWorkersForDevice:util-runmodes.c:254]
Config: flow-manager: using 1 flow manager threads [FlowManagerThreadSpawn:flow-manager.c:959]
Config: flow-manager: using 1 flow recycler threads [FlowRecyclerThreadSpawn:flow-manager.c:1162]
Info: unix-manager: unix socket '/var/run/suricata/suricata-command.socket' [UnixNew:unix-manager.c:136]
Perf: af-packet: eno1: setting socket buffer to 64535 [AFPCreateSocket:source-af-packet.c:1950]
Perf: af-packet: eno1: rx ring: block_size=32768 block_nr=10001 frame_size=1600 frame_nr=200020 [AFPComputeRingParams:source-af-packet.c:1600]
Perf: af-packet: eno2: setting socket buffer to 64535 [AFPCreateSocket:source-af-packet.c:1950]
Perf: af-packet: eno2: rx ring: block_size=32768 block_nr=10001 frame_size=1600 frame_nr=200020 [AFPComputeRingParams:source-af-packet.c:1600]
Notice: threads: Threads created -> W: 2 FM: 1 FR: 1   Engine started. [TmThreadWaitOnThreadRunning:tm-threads.c:1891]

open another ssh session to run xdp-loader status to show the xdp program is attached ok

root@r210:~# xdp-loader status
CURRENT XDP PROGRAM STATUS:

Interface        Prio  Program name      Mode     ID   Tag               Chain actions
--------------------------------------------------------------------------------------
lo                     <No XDP program loaded!>
enp2s0f0               <No XDP program loaded!>
enp2s0f1               <No XDP program loaded!>
eno1                   xdp_hashfilter    native   319  f1787f7950a1278f 
eno2                   xdp_hashfilter    native   322  f1787f7950a1278f 
br0                    <No XDP program loaded!>
wg0                    <No XDP program loaded!>
virbr0                 <No XDP program loaded!>

then run xdp-loader to detach the program manually

root@r210:~# xdp-loader unload eno1 -a
root@r210:~# xdp-loader status
CURRENT XDP PROGRAM STATUS:

Interface        Prio  Program name      Mode     ID   Tag               Chain actions
--------------------------------------------------------------------------------------
lo                     <No XDP program loaded!>
enp2s0f0               <No XDP program loaded!>
enp2s0f1               <No XDP program loaded!>
eno1                   <No XDP program loaded!>
eno2                   xdp_hashfilter    native   322  f1787f7950a1278f 
br0                    <No XDP program loaded!>
wg0                    <No XDP program loaded!>
virbr0                 <No XDP program loaded!>
go back to the ssh session window where Suricata is started, control + c to stop Suricata
^CNotice: suricata: Signal Received.  Stopping engine. [SuricataMainLoop:suricata.c:2811]
Info: suricata: time elapsed 460.567s [SCPrintElapsedTime:suricata.c:1167]
Perf: flow-manager: 29 flows processed [FlowRecycler:flow-manager.c:1134]
Perf: af-packet: eno1: (W#01-eno1) kernel: Packets 28, dropped 0 [ReceiveAFPThreadExitStats:source-af-packet.c:2631]
Perf: af-packet: eno2: (W#01-eno2) kernel: Packets 40, dropped 0 [ReceiveAFPThreadExitStats:source-af-packet.c:2631]
Info: counters: Alerts: 0 [StatsLogSummary:counters.c:888]
Perf: ippair: ippair memory usage: 406144 bytes, maximum: 16777216 [IPPairPrintStats:ippair.c:295]
Perf: host: host memory usage: 390144 bytes, maximum: 33554432 [HostPrintStats:host.c:298]
Perf: app-layer-htp: htp memory 0 (0) [AppLayerHtpPrintStats:app-layer-htp.c:3036]
Notice: device: eno1: packets: 28, drops: 0 (0.00%), invalid chksum: 0 [LiveDeviceListClean:util-device.c:331]
Notice: device: eno2: packets: 40, drops: 0 (0.00%), invalid chksum: 0 [LiveDeviceListClean:util-device.c:331]

start Suricata again:

Info: af-packet: eno1: AF_PACKET IPS mode activated eno1->eno2 [ParseAFPConfig:runmode-af-packet.c:353]
Config: af-packet: eno1: using queue based cluster mode for AF_PACKET [ParseAFPConfig:runmode-af-packet.c:410]
Config: af-packet: eno1: using pinned maps [ParseAFPConfig:runmode-af-packet.c:454]
Info: ebpf: Loaded pinned maps, will use already loaded eBPF filter [EBPFLoadFile:util-ebpf.c:317]
Info: af-packet: eno1: loaded pinned maps from sysfs [ParseAFPConfig:runmode-af-packet.c:596]
Info: ioctl: eno1: RX RSS queues: 8 [GetIfaceRSSQueuesNum:util-ioctl.c:734]
Perf: af-packet: eno1: cluster_qm: 8 RSS queues, using 8 threads [ParseAFPConfig:runmode-af-packet.c:727]
Info: runmodes: eno1: creating 1 thread [RunModeSetLiveCaptureWorkersForDevice:util-runmodes.c:254]
Info: af-packet: eno2: AF_PACKET IPS mode activated eno2->eno1 [ParseAFPConfig:runmode-af-packet.c:353]
Config: af-packet: eno2: using queue based cluster mode for AF_PACKET [ParseAFPConfig:runmode-af-packet.c:410]
Config: af-packet: eno2: using pinned maps [ParseAFPConfig:runmode-af-packet.c:454]
Info: ebpf: Loaded pinned maps, will use already loaded eBPF filter [EBPFLoadFile:util-ebpf.c:317]
Info: af-packet: eno2: loaded pinned maps from sysfs [ParseAFPConfig:runmode-af-packet.c:596]
Info: ioctl: eno2: RX RSS queues: 8 [GetIfaceRSSQueuesNum:util-ioctl.c:734]
Perf: af-packet: eno2: cluster_qm: 8 RSS queues, using 8 threads [ParseAFPConfig:runmode-af-packet.c:727]
Info: runmodes: eno2: creating 1 thread [RunModeSetLiveCaptureWorkersForDevice:util-runmodes.c:254]
Config: flow-manager: using 1 flow manager threads [FlowManagerThreadSpawn:flow-manager.c:959]
Config: flow-manager: using 1 flow recycler threads [FlowRecyclerThreadSpawn:flow-manager.c:1162]
Info: unix-manager: unix socket '/var/run/suricata/suricata-command.socket' [UnixNew:unix-manager.c:136]
Perf: af-packet: eno1: setting socket buffer to 64535 [AFPCreateSocket:source-af-packet.c:1950]
Perf: af-packet: eno1: rx ring: block_size=32768 block_nr=10001 frame_size=1600 frame_nr=200020 [AFPComputeRingParams:source-af-packet.c:1600]
Perf: af-packet: eno2: setting socket buffer to 64535 [AFPCreateSocket:source-af-packet.c:1950]
Perf: af-packet: eno2: rx ring: block_size=32768 block_nr=10001 frame_size=1600 frame_nr=200020 [AFPComputeRingParams:source-af-packet.c:1600]
Notice: threads: Threads created -> W: 2 FM: 1 FR: 1   Engine started. [TmThreadWaitOnThreadRunning:tm-threads.c:1891]

xdp-loader status shows xdp program is not attached to eno1 interface

root@r210:~# xdp-loader status
CURRENT XDP PROGRAM STATUS:

Interface        Prio  Program name      Mode     ID   Tag               Chain actions
--------------------------------------------------------------------------------------
lo                     <No XDP program loaded!>
enp2s0f0               <No XDP program loaded!>
enp2s0f1               <No XDP program loaded!>
eno1                   <No XDP program loaded!>
eno2                   xdp_hashfilter    native   322  f1787f7950a1278f 
br0                    <No XDP program loaded!>
wg0                    <No XDP program loaded!>
virbr0                 <No XDP program loaded!>

the reason is EBPFLoadFile will not open/load/attach XDP program when pinned-maps is set to true

299 int EBPFLoadFile(const char *iface, const char *path, const char * section,
 300                  int *val, struct ebpf_timeout_config *config)
 301 {
 302     int err, pfd;
 303     bool found = false;
 304     struct bpf_object *bpfobj = NULL;
 305     struct bpf_program *bpfprog = NULL;
 306     struct bpf_map *map = NULL;
 307 
 308     if (iface == NULL)
 309         return -1;
 310     LiveDevice *livedev = LiveGetDevice(iface);
 311     if (livedev == NULL)
 312         return -1;
 313 
 314     if (config->flags & EBPF_XDP_CODE && config->flags & EBPF_PINNED_MAPS) { <====================
 315         /* We try to get our flow table maps and if we have them we can simply return */
 316         if (EBPFLoadPinnedMaps(livedev, config) == 0) {.   <========================================
 317             SCLogInfo("Loaded pinned maps, will use already loaded eBPF filter");
 318             return 1;
 319         }
 320     }

I know this is corner case that users are not suppose to unload the XDP program manually with other xdp utility program, but for whatever unexpected situation, if the XDP program is detached, when suricata is started, suricata should check if program is attached when pinned-maps is set to true.

Actions

Also available in: Atom PDF