Project

General

Profile

Actions

Bug #2842

closed

IPS mode crash under load

Added by Ad Schellevis almost 4 years ago. Updated almost 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

When using suricata 4.1.2 in netmap IPS mode on FreeBSD and performing a stress test using Trex, suricata stops handling traffic and core dumps later.
The issue doesn't appear to exist on 4.0.x, we performed the same test on the same hardware and operating system without noticeable issues.

At first suricata peaks at high cpu load (for a short while), then drops quite rapidly and no traffic is being forwarded anymore. After a minute or so (while still receiving traffic), suricata core dumps (details below)

------------------------------
Device under test:
------------------------------
os : OPNsense 19.1.1
cpu : AMD GX-420MC SOC
nic 1 [igb0] Intel i210 : 10.0.0.1/24
nic 2 [igb1] Intel i210 : 10.1.0.1/24

--------------
Test setup
--------------
Trex connected using 10.0.0.2, 10.1.0.2 communicating from/to (bidirectional) 16.0.0.0/8 <-> 48.0.0.0/8 using an ASTFProfile profile class generating different real traffic packets (http. https. smtp, ...) of various sizes.

----------------------
suricata --build-info
----------------------

This is Suricata version 4.1.2 RELEASE
Features: IPFW PCAP_SET_BUFF NETMAP HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_LIBJANSSON TLS MAGIC 
SIMD support: SSE_4_2 SSE_4_1 SSE_3 
Atomic intrisics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 4.2.1 Compatible FreeBSD Clang 6.0.0 (tags/RELEASE_600/final 326565), C version 199901
compiled with _FORTIFY_SOURCE=0
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.29, linked against LibHTP v0.5.29

Suricata Configuration:
  AF_PACKET support:                       no
  eBPF support:                            no
  XDP support:                             no
  PF_RING support:                         no
  NFQueue support:                         no
  NFLOG support:                           no
  IPFW support:                            yes
  Netmap support:                          yes
  DAG enabled:                             no
  Napatech enabled:                        no
  WinDivert enabled:                       no

  Unix socket enabled:                     yes
  Detection enabled:                       yes

  Libmagic support:                        yes
  libnss support:                          no
  libnspr support:                         no
  libjansson support:                      yes
  liblzma support:                         yes
  hiredis support:                         no
  hiredis async with libevent:             no
  Prelude support:                         no
  PCRE jit:                                yes
  LUA support:                             no
  libluajit:                               no
  libgeoip:                                no
  Non-bundled htp:                         no
  Old barnyard2 support:                   no
  Hyperscan support:                       yes
  Libnet support:                          yes
  liblz4 support:                          yes

  Rust support:                            no
  Rust strict mode:                        no
  Rust debug mode:                         no
  Rust compiler:                           not set
  Rust cargo:                              not set

  Install suricatasc:                      yes
  Install suricata-update:                 yes

  Profiling enabled:                       no
  Profiling locks enabled:                 no

Development settings:
  Coccinelle / spatch:                     no
  Unit tests enabled:                      no
  Debug output enabled:                    no
  Debug validation enabled:                no

Generic build parameters:
  Installation prefix:                     /usr/local
  Configuration directory:                 /usr/local/etc/suricata/
  Log directory:                           /var/log/suricata/

  --prefix                                 /usr/local
  --sysconfdir                             /usr/local/etc
  --localstatedir                          /var
  --datarootdir                            /usr/local/share

  Host:                                    x86_64-unknown-freebsd11.2
  Compiler:                                cc (exec name) / clang (real)
  GCC Protect enabled:                     no
  GCC march native enabled:                yes
  GCC Profile enabled:                     no
  Position Independent Executable enabled: no
  CFLAGS                                   -ggdb -O0  -pipe  -DHARDENEDBSD -fsanitize=safe-stack -fstack-protector-all -fno-strict-aliasing  -DOS_FREEBSD -DOS_FREEBSD -march=native
  PCAP_CFLAGS                              
  SECCFLAGS                                

------------------
output of gdb
------------------

# gdb /usr/local/bin/suricata  suricata.core 
GNU gdb (GDB) 8.2 [GDB v8.2 for FreeBSD]
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd11.2".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/bin/suricata...done.
[New LWP 100544]
[New LWP 100206]
[New LWP 100541]
[New LWP 100543]
[New LWP 100545]
[New LWP 100546]
[New LWP 100547]
[New LWP 100548]
[New LWP 100549]
[New LWP 100550]
Core was generated by `/usr/local/bin/suricata --netmap --pidfile /var/run/suricata.pid -c /usr/local/e'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000745b6c in SCACSearch (mpm_ctx=0x31162d0c330, mpm_thread_ctx=0x3117fdb7128, pmq=0x3117fdb7138, buf=0x31286776cef <error: Cannot access memory at address 0x31286776cef>, buflen=2636)
    at util-mpm-ac.c:1048
1048                state = state_table_u16[state & 0x7FFF][u8_tolower(buf[i])];
[Current thread is 1 (LWP 100544)]
(gdb) set logging on
Copying output to gdb.txt.
(gdb) thread apply all bt

Thread 10 (LWP 100550):
#0  0x0000031115b29b5c in ?? () from /lib/libthr.so.3
#1  0x0000031115b1dbf1 in ?? () from /lib/libthr.so.3
#2  0x0000031115b27888 in ?? () from /lib/libthr.so.3
#3  0x00000000004e6744 in StatsMgmtThread (arg=0x3115c362f00) at counters.c:392
#4  0x0000031115b1bc36 in ?? () from /lib/libthr.so.3
#5  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x6bc326e7a000

Thread 9 (LWP 100549):
#0  0x0000031115b29b5c in ?? () from /lib/libthr.so.3
#1  0x0000031115b1dbf1 in ?? () from /lib/libthr.so.3
#2  0x0000031115b27888 in ?? () from /lib/libthr.so.3
#3  0x00000000004e5f71 in StatsWakeupThread (arg=0x3115c362e00) at counters.c:467
#4  0x0000031115b1bc36 in ?? () from /lib/libthr.so.3
#5  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x6bc32707b000

Thread 8 (LWP 100548):
#0  0x0000031115b29b5c in ?? () from /lib/libthr.so.3
#1  0x0000031115b1dbf1 in ?? () from /lib/libthr.so.3
#2  0x0000031115b27888 in ?? () from /lib/libthr.so.3
--Type <RET> for more, q to quit, c to continue without paging--c
#3  0x00000000006001c5 in FlowRecycler (th_v=0x3115c362d00, thread_data=0x3118200d000) at flow-manager.c:940
#4  0x00000000006f3cdd in TmThreadsManagement (td=0x3115c362d00) at tm-threads.c:719
#5  0x0000031115b1bc36 in ?? () from /lib/libthr.so.3
#6  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x6bc32727c000

Thread 7 (LWP 100547):
#0  0x0000031115b29b5c in ?? () from /lib/libthr.so.3
#1  0x0000031115b1dbf1 in ?? () from /lib/libthr.so.3
#2  0x0000031115b27888 in ?? () from /lib/libthr.so.3
#3  0x00000000005ff7d8 in FlowManager (th_v=0x3115c362c00, thread_data=0x3118180d000) at flow-manager.c:787
#4  0x00000000006f3cdd in TmThreadsManagement (td=0x3115c362c00) at tm-threads.c:719
#5  0x0000031115b1bc36 in ?? () from /lib/libthr.so.3
#6  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x6bc32747d000

Thread 6 (LWP 100546):
#0  0x000000000067c671 in OutputTxLog (tv=0x311810b7600, p=0x31180ffe2f0, thread_data=0x31180ffe2f0) at output-tx.c:131
#1  0x000000000063a228 in OutputLoggerLog (tv=0x3115c362b00, p=0x311810b7600, thread_data=0x31180ffe2f0) at output.c:921
#2  0x0000000000605998 in FlowWorker (tv=0x3115c362b00, p=0x311810b7600, data=0x311810da000, preq=0x3115ffba780, unused=0x0) at flow-worker.c:265
#3  0x00000000006ec067 in TmThreadsSlotVarRun (tv=0x3115c362b00, p=0x311810b7600, slot=0x3115ffba680) at tm-threads.c:145
#4  0x00000000006a9dd4 in TmThreadsSlotProcessPkt (tv=0x3115c362b00, s=0x3115ffba680, p=0x311810b7600) at ./tm-threads.h:147
#5  0x00000000006a9c68 in NetmapRingRead (ntv=0x311810bd000, ring_id=4) at source-netmap.c:816
#6  0x00000000006a767c in ReceiveNetmapLoop (tv=0x3115c362b00, data=0x311810bd000, slot=0x3115ffba5c0) at source-netmap.c:896
#7  0x00000000006f3739 in TmThreadsSlotPktAcqLoop (td=0x3115c362b00) at tm-threads.c:348
#8  0x0000031115b1bc36 in ?? () from /lib/libthr.so.3
#9  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x6bc32767e000

Thread 5 (LWP 100545):
#0  0x00000000006cf273 in TCPChecksum (shdr=0x3116c28c01a, pkt=0x3116c28c336, tlen=692, init=49835) at ./decode-tcp.h:197
#1  0x00000000006bd5b5 in StreamTcpValidateChecksum (p=0x311806b7600) at stream-tcp.c:4917
#2  0x00000000006bd456 in StreamTcp (tv=0x3115c362900, p=0x311806b7600, data=0x311806da050, pq=0x311806da020, postpq=0x0) at stream-tcp.c:5114
#3  0x0000000000605820 in FlowWorker (tv=0x3115c362900, p=0x311806b7600, data=0x311806da000, preq=0x3115ffba480, unused=0x0) at flow-worker.c:216
#4  0x00000000006ec067 in TmThreadsSlotVarRun (tv=0x3115c362900, p=0x311806b7600, slot=0x3115ffba380) at tm-threads.c:145
#5  0x00000000006a9dd4 in TmThreadsSlotProcessPkt (tv=0x3115c362900, s=0x3115ffba380, p=0x311806b7600) at ./tm-threads.h:147
#6  0x00000000006a9c68 in NetmapRingRead (ntv=0x311806bd000, ring_id=3) at source-netmap.c:816
#7  0x00000000006a767c in ReceiveNetmapLoop (tv=0x3115c362900, data=0x311806bd000, slot=0x3115ffba2c0) at source-netmap.c:896
#8  0x00000000006f3739 in TmThreadsSlotPktAcqLoop (td=0x3115c362900) at tm-threads.c:348
#9  0x0000031115b1bc36 in ?? () from /lib/libthr.so.3
#10 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x6bc32787f000

Thread 4 (LWP 100543):
#0  0x0000000000745b94 in SCACSearch (mpm_ctx=0x31162d0c330, mpm_thread_ctx=0x3117f3b7108, pmq=0x3117f3b7138, buf=0x3116c2bc042 <error: Cannot access memory at address 0x3116c2bc042>, buflen=1448) at util-mpm-ac.c:1049
#1  0x00000000005751de in PrefilterPktStream (det_ctx=0x3117f3b7000, p=0x3117f2b7600, pectx=0x31162d0c330) at detect-engine-payload.c:102
#2  0x000000000057adf3 in Prefilter (det_ctx=0x3117f3b7000, sgh=0x31162d7b7a0, p=0x3117f2b7600, flags=8 '\b') at detect-engine-prefilter.c:169
#3  0x000000000051626a in DetectRunPrefilterPkt (tv=0x3115c362700, de_ctx=0x3115b0c7000, det_ctx=0x3117f3b7000, p=0x3117f2b7600, scratch=0x3117f11df90) at detect.c:733
#4  0x00000000005158d2 in DetectRun (th_v=0x3115c362700, de_ctx=0x3115b0c7000, det_ctx=0x3117f3b7000, p=0x3117f2b7600) at detect.c:131
#5  0x0000000000515527 in DetectFlow (tv=0x3115c362700, de_ctx=0x3115b0c7000, det_ctx=0x3117f3b7000, p=0x3117f2b7600) at detect.c:1661
#6  0x0000000000515362 in Detect (tv=0x3115c362700, p=0x3117f2b7600, data=0x3117f3b7000, pq=0x0, postpq=0x0) at detect.c:1735
#7  0x000000000060597d in FlowWorker (tv=0x3115c362700, p=0x3117f2b7600, data=0x3117f2da000, preq=0x3115ffb9e80, unused=0x0) at flow-worker.c:260
#8  0x00000000006ec067 in TmThreadsSlotVarRun (tv=0x3115c362700, p=0x3117f2b7600, slot=0x3115ffb9d80) at tm-threads.c:145
#9  0x00000000006a9dd4 in TmThreadsSlotProcessPkt (tv=0x3115c362700, s=0x3115ffb9d80, p=0x3117f2b7600) at ./tm-threads.h:147
#10 0x00000000006a9c68 in NetmapRingRead (ntv=0x3117f2bd000, ring_id=1) at source-netmap.c:816
#11 0x00000000006a767c in ReceiveNetmapLoop (tv=0x3115c362700, data=0x3117f2bd000, slot=0x3115ffb9cc0) at source-netmap.c:896
#12 0x00000000006f3739 in TmThreadsSlotPktAcqLoop (td=0x3115c362700) at tm-threads.c:348
#13 0x0000031115b1bc36 in ?? () from /lib/libthr.so.3
#14 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x6bc327c81000

Thread 3 (LWP 100541):
#0  0x000000000074a3c1 in __getCurrentRuneLocale () at /usr/include/runetype.h:94
#1  0x0000000000746375 in __sbtolower (_c=42) at /usr/include/_ctype.h:155
#2  0x0000000000745b7c in SCACSearch (mpm_ctx=0x31162d0c330, mpm_thread_ctx=0x31165db8108, pmq=0x31165db8138, buf=0x3116b328042 <error: Cannot access memory at address 0x3116b328042>, buflen=1448) at util-mpm-ac.c:1048
#3  0x00000000005751de in PrefilterPktStream (det_ctx=0x31165db8000, p=0x31165cb7600, pectx=0x31162d0c330) at detect-engine-payload.c:102
#4  0x000000000057adf3 in Prefilter (det_ctx=0x31165db8000, sgh=0x31162d79950, p=0x31165cb7600, flags=8 '\b') at detect-engine-prefilter.c:169
#5  0x000000000051626a in DetectRunPrefilterPkt (tv=0x3115c362600, de_ctx=0x3115b0c7000, det_ctx=0x31165db8000, p=0x31165cb7600, scratch=0x311667fcf90) at detect.c:733
#6  0x00000000005158d2 in DetectRun (th_v=0x3115c362600, de_ctx=0x3115b0c7000, det_ctx=0x31165db8000, p=0x31165cb7600) at detect.c:131
#7  0x0000000000515527 in DetectFlow (tv=0x3115c362600, de_ctx=0x3115b0c7000, det_ctx=0x31165db8000, p=0x31165cb7600) at detect.c:1661
#8  0x0000000000515362 in Detect (tv=0x3115c362600, p=0x31165cb7600, data=0x31165db8000, pq=0x0, postpq=0x0) at detect.c:1735
#9  0x000000000060597d in FlowWorker (tv=0x3115c362600, p=0x31165cb7600, data=0x31165cdb000, preq=0x3115ffb9b80, unused=0x0) at flow-worker.c:260
#10 0x00000000006ec067 in TmThreadsSlotVarRun (tv=0x3115c362600, p=0x31165cb7600, slot=0x3115ffb9a80) at tm-threads.c:145
#11 0x00000000006a9dd4 in TmThreadsSlotProcessPkt (tv=0x3115c362600, s=0x3115ffb9a80, p=0x31165cb7600) at ./tm-threads.h:147
#12 0x00000000006a9c68 in NetmapRingRead (ntv=0x31165cbd000, ring_id=0) at source-netmap.c:816
#13 0x00000000006a767c in ReceiveNetmapLoop (tv=0x3115c362600, data=0x31165cbd000, slot=0x3115ffb99c0) at source-netmap.c:896
#14 0x00000000006f3739 in TmThreadsSlotPktAcqLoop (td=0x3115c362600) at tm-threads.c:348
#15 0x0000031115b1bc36 in ?? () from /lib/libthr.so.3
#16 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x6bc327e82000

Thread 2 (LWP 100206):
#0  0x0000031116fa85da in _nanosleep () from /lib/libc.so.7
#1  0x0000031115b1e63c in ?? () from /lib/libthr.so.3
#2  0x0000031116ff98c6 in usleep () from /lib/libc.so.7
#3  0x00000000006e2f49 in SuricataMainLoop (suri=0xc78cd8 <suricata>) at suricata.c:2883
#4  0x00000000006db8b4 in main (argc=6, argv=0x6bc367e82948) at suricata.c:3026

Thread 1 (LWP 100544):
#0  0x0000000000745b6c in SCACSearch (mpm_ctx=0x31162d0c330, mpm_thread_ctx=0x3117fdb7128, pmq=0x3117fdb7138, buf=0x31286776cef <error: Cannot access memory at address 0x31286776cef>, buflen=2636) at util-mpm-ac.c:1048
#1  0x0000000000575c2a in StreamMpmFunc (cb_data=0x3117f7fcf78, data=0x31286776cef <error: Cannot access memory at address 0x31286776cef>, data_len=2636) at detect-engine-payload.c:64
#2  0x00000000006d66eb in StreamReassembleRawInline (ssn=0x3118622cc00, p=0x3117fcb7600, Callback=0x575b90 <StreamMpmFunc>, cb_data=0x3117f7fcf78, progress_out=0x3117fdb7050) at stream-tcp-reassemble.c:1470
#3  0x00000000006d5f2c in StreamReassembleRaw (ssn=0x3118622cc00, p=0x3117fcb7600, Callback=0x575b90 <StreamMpmFunc>, cb_data=0x3117f7fcf78, progress_out=0x3117fdb7050, respect_inspect_depth=false) at stream-tcp-reassemble.c:1660
#4  0x000000000057512a in PrefilterPktStream (det_ctx=0x3117fdb7000, p=0x3117fcb7600, pectx=0x31162d0c330) at detect-engine-payload.c:83
#5  0x000000000057adf3 in Prefilter (det_ctx=0x3117fdb7000, sgh=0x31162d78c30, p=0x3117fcb7600, flags=136 '\210') at detect-engine-prefilter.c:169
#6  0x000000000051626a in DetectRunPrefilterPkt (tv=0x3115c362800, de_ctx=0x3115b0c7000, det_ctx=0x3117fdb7000, p=0x3117fcb7600, scratch=0x3117f7fcf90) at detect.c:733
#7  0x00000000005158d2 in DetectRun (th_v=0x3115c362800, de_ctx=0x3115b0c7000, det_ctx=0x3117fdb7000, p=0x3117fcb7600) at detect.c:131
#8  0x0000000000515527 in DetectFlow (tv=0x3115c362800, de_ctx=0x3115b0c7000, det_ctx=0x3117fdb7000, p=0x3117fcb7600) at detect.c:1661
#9  0x0000000000515362 in Detect (tv=0x3115c362800, p=0x3117fcb7600, data=0x3117fdb7000, pq=0x0, postpq=0x0) at detect.c:1735
#10 0x000000000060597d in FlowWorker (tv=0x3115c362800, p=0x3117fcb7600, data=0x3117fcda000, preq=0x3115ffba180, unused=0x0) at flow-worker.c:260
#11 0x00000000006ec067 in TmThreadsSlotVarRun (tv=0x3115c362800, p=0x3117fcb7600, slot=0x3115ffba080) at tm-threads.c:145
#12 0x00000000006a9dd4 in TmThreadsSlotProcessPkt (tv=0x3115c362800, s=0x3115ffba080, p=0x3117fcb7600) at ./tm-threads.h:147
#13 0x00000000006a9c68 in NetmapRingRead (ntv=0x3117fcbd000, ring_id=2) at source-netmap.c:816
#14 0x00000000006a767c in ReceiveNetmapLoop (tv=0x3115c362800, data=0x3117fcbd000, slot=0x3115ffb9fc0) at source-netmap.c:896
#15 0x00000000006f3739 in TmThreadsSlotPktAcqLoop (td=0x3115c362800) at tm-threads.c:348
#16 0x0000031115b1bc36 in ?? () from /lib/libthr.so.3
#17 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x6bc327a80000


Related issues 1 (0 open1 closed)

Copied to Bug #2846: IPS mode crash under load (5.0.x)ClosedVictor JulienActions
Actions #1

Updated by Victor Julien almost 4 years ago

  • Subject changed from Netmap mode crash under load / FreeBSD to IPS mode crash under load
  • Status changed from New to Assigned
  • Assignee set to Victor Julien
  • Target version set to 4.1.3

Confirmed and testing fix. Issue is caused by memory pressure (memcap) on stream engine running in 'inline' mode.

Actions #2

Updated by Victor Julien almost 4 years ago

  • Copied to Bug #2846: IPS mode crash under load (5.0.x) added
Actions #3

Updated by Victor Julien almost 4 years ago

  • Status changed from Assigned to Closed
  • Priority changed from High to Normal
  • Affected Versions 4.1, 4.1.1 added
Actions

Also available in: Atom PDF