Project

General

Profile

Actions

Bug #4080

closed

DCERPCUDPState handle fragmented data functions pegging certain CPU cores/threads

Added by Peter Manev 11 months ago. Updated 7 months ago.

Status:
Closed
Priority:
High
Target version:
Affected Versions:
Effort:
Difficulty:
Label:
Needs backport to 6.0

Description

Noticed on live runs - during certain periods i get some CPUs pegged to 100% then after some time (10-15) min the "pegging" disappears.
When chasing it around i compiled with debugging enabled and while that by itself has a perf impact, it is visible that the DCERPCUDPState handle fragmented data functions are way up in CPU usage. Please see attached.

Using:

/opt/suritest-tmp/bin/suricata --build-info
This is Suricata version 6.0.1-dev (95729e923 2020-10-09)
Features: PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LUA HAVE_LUAJIT HAVE_LIBJANSSON TLS TLS_C11 MAGIC RUST 
SIMD support: SSE_4_2 SSE_4_1 SSE_3 
Atomic intrinsics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 4.2.1 Compatible Clang 7.0.1 (tags/RELEASE_701/final), C version 201112
compiled with _FORTIFY_SOURCE=0
L1 cache line size (CLS)=64
thread local storage method: _Thread_local
compiled with LibHTP v0.5.35, linked against LibHTP v0.5.35

Suricata Configuration:
  AF_PACKET support:                       yes
  eBPF support:                            yes
  XDP support:                             yes
  PF_RING support:                         no
  NFQueue support:                         no
  NFLOG support:                           no
  IPFW support:                            no
  Netmap support:                          no 
  DAG enabled:                             no
  Napatech enabled:                        no
  WinDivert enabled:                       no

  Unix socket enabled:                     yes
  Detection enabled:                       yes

  Libmagic support:                        yes
  libnss support:                          yes
  libnspr support:                         yes
  libjansson support:                      yes
  hiredis support:                         no
  hiredis async with libevent:             no
  Prelude support:                         no
  PCRE jit:                                yes
  LUA support:                             yes, through luajit
  libluajit:                               yes
  GeoIP2 support:                          yes
  Non-bundled htp:                         no
  Hyperscan support:                       yes
  Libnet support:                          yes
  liblz4 support:                          yes

  Rust support:                            yes
  Rust strict mode:                        yes
  Rust compiler path:                      /root/.cargo/bin/rustc
  Rust compiler version:                   rustc 1.47.0 (18bf6b4f0 2020-10-07)
  Cargo path:                              /root/.cargo/bin/cargo
  Cargo version:                           cargo 1.47.0 (f3c7e066a 2020-08-28)
  Cargo vendor:                            yes

  Python support:                          yes
  Python path:                             /usr/bin/python3
  Python distutils                         yes
  Python yaml                              yes
  Install suricatactl:                     yes
  Install suricatasc:                      yes
  Install suricata-update:                 not bundled

  Profiling enabled:                       no
  Profiling locks enabled:                 no

  Plugin support (experimental):           yes

Development settings:
  Coccinelle / spatch:                     no
  Unit tests enabled:                      no
  Debug output enabled:                    no
  Debug validation enabled:                no

Generic build parameters:
  Installation prefix:                     /opt/suritest-tmp
  Configuration directory:                 /opt/suritest-tmp/etc/suricata/
  Log directory:                           /opt/suritest-tmp/var/log/suricata/

  --prefix                                 /opt/suritest-tmp
  --sysconfdir                             /opt/suritest-tmp/etc
  --localstatedir                          /opt/suritest-tmp/var
  --datarootdir                            /opt/suritest-tmp/share

  Host:                                    x86_64-pc-linux-gnu
  Compiler:                                clang (exec name) / g++ (real)
  GCC Protect enabled:                     no
  GCC march native enabled:                yes
  GCC Profile enabled:                     no
  Position Independent Executable enabled: no
  CFLAGS                                   -ggdb3 -O0  -Wchar-subscripts -Wshadow -Wall -Wextra -Wno-unused-parameter -Wno-unused-function  -Wno-unused-parameter -Wno-unused-function -std=c11 -march=native -I${srcdir}/../rust/gen -I${srcdir}/../rust/dist
  PCAP_CFLAGS                               -I/usr/include
  SECCFLAGS                                


Files


Related issues

Related to Bug #4124: dcerpc: UDP request response pair match is incorrectClosedCommunity TicketActions
Copied to Bug #4315: DCERPCUDPState handle fragmented data functions pegging certain CPU cores/threads ClosedShivani BhardwajActions
Actions #1

Updated by Victor Julien 10 months ago

  • Related to Bug #4124: dcerpc: UDP request response pair match is incorrect added
Actions #2

Updated by Peter Manev 10 months ago

With the latest code merges - this is in live testing currently.
(7b4ac8dba 2020-11-19)

Actions #3

Updated by Peter Manev 10 months ago

Still seeing this. trying a different investigation approach.

-   46.94%    46.94%  suricata            [.] suricata::dcerpc::dcerpc_udp::DCERPCUDPState::handle_fragment_data
     38.97% suricata::dcerpc::dcerpc_udp::DCERPCUDPState::handle_fragment_data
-   44.65%     0.06%  suricata            [.] AFPWalkBlock
     7.22% AFPWalkBlock
-   44.57%     0.52%  suricata            [.] AFPParsePacketV3
   - 6.93% AFPParsePacketV3
      - 6.94% TmThreadsSlotProcessPkt
         - 7.19% TmThreadsSlotVarRun
            - 7.01% FlowWorker
               - 3.96% Detect
                  - 4.00% DetectFlow
                     - 5.57% DetectRun
                        - 3.73% DetectRunPrefilterPkt
                           - 3.11% Prefilter
                              - 2.38% PrefilterPktPayload
                                 - 2.35% SCHSSearch
                                    - 2.60% avx512_hs_scan
                                       - 5.37% 0x7f7bb8b5cb85
                                            2.06% 0x7f7bb8b24ee6
                                            0.91% 0x7f7bb8b24f36
                                       - 1.39% 0x7f7bb8b5cb0c
                                          - 5.61% 0x7f7bb8a6da21

Actions #4

Updated by Shivani Bhardwaj 8 months ago

  • Status changed from New to Assigned
  • Assignee set to Shivani Bhardwaj
  • Priority changed from Normal to High
  • Target version set to 6.0.2
Actions #5

Updated by Victor Julien 8 months ago

  • Target version changed from 6.0.2 to 7.0rc1
  • Label Needs backport to 6.0 added
Actions #6

Updated by Jeff Lucovsky 8 months ago

  • Copied to Bug #4315: DCERPCUDPState handle fragmented data functions pegging certain CPU cores/threads added
Actions #8

Updated by Shivani Bhardwaj 7 months ago

  • Status changed from Assigned to Closed
Actions

Also available in: Atom PDF