Actions
Bug #2842
closedIPS mode crash under load
Description
When using suricata 4.1.2 in netmap IPS mode on FreeBSD and performing a stress test using Trex, suricata stops handling traffic and core dumps later.
The issue doesn't appear to exist on 4.0.x, we performed the same test on the same hardware and operating system without noticeable issues.
At first suricata peaks at high cpu load (for a short while), then drops quite rapidly and no traffic is being forwarded anymore. After a minute or so (while still receiving traffic), suricata core dumps (details below)
------------------------------
Device under test:
------------------------------
os : OPNsense 19.1.1
cpu : AMD GX-420MC SOC
nic 1 [igb0] Intel i210 : 10.0.0.1/24
nic 2 [igb1] Intel i210 : 10.1.0.1/24
--------------
Test setup
--------------
Trex connected using 10.0.0.2, 10.1.0.2 communicating from/to (bidirectional) 16.0.0.0/8 <-> 48.0.0.0/8 using an ASTFProfile profile class generating different real traffic packets (http. https. smtp, ...) of various sizes.
----------------------
suricata --build-info
----------------------
This is Suricata version 4.1.2 RELEASE Features: IPFW PCAP_SET_BUFF NETMAP HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_LIBJANSSON TLS MAGIC SIMD support: SSE_4_2 SSE_4_1 SSE_3 Atomic intrisics: 1 2 4 8 16 byte(s) 64-bits, Little-endian architecture GCC version 4.2.1 Compatible FreeBSD Clang 6.0.0 (tags/RELEASE_600/final 326565), C version 199901 compiled with _FORTIFY_SOURCE=0 L1 cache line size (CLS)=64 thread local storage method: __thread compiled with LibHTP v0.5.29, linked against LibHTP v0.5.29 Suricata Configuration: AF_PACKET support: no eBPF support: no XDP support: no PF_RING support: no NFQueue support: no NFLOG support: no IPFW support: yes Netmap support: yes DAG enabled: no Napatech enabled: no WinDivert enabled: no Unix socket enabled: yes Detection enabled: yes Libmagic support: yes libnss support: no libnspr support: no libjansson support: yes liblzma support: yes hiredis support: no hiredis async with libevent: no Prelude support: no PCRE jit: yes LUA support: no libluajit: no libgeoip: no Non-bundled htp: no Old barnyard2 support: no Hyperscan support: yes Libnet support: yes liblz4 support: yes Rust support: no Rust strict mode: no Rust debug mode: no Rust compiler: not set Rust cargo: not set Install suricatasc: yes Install suricata-update: yes Profiling enabled: no Profiling locks enabled: no Development settings: Coccinelle / spatch: no Unit tests enabled: no Debug output enabled: no Debug validation enabled: no Generic build parameters: Installation prefix: /usr/local Configuration directory: /usr/local/etc/suricata/ Log directory: /var/log/suricata/ --prefix /usr/local --sysconfdir /usr/local/etc --localstatedir /var --datarootdir /usr/local/share Host: x86_64-unknown-freebsd11.2 Compiler: cc (exec name) / clang (real) GCC Protect enabled: no GCC march native enabled: yes GCC Profile enabled: no Position Independent Executable enabled: no CFLAGS -ggdb -O0 -pipe -DHARDENEDBSD -fsanitize=safe-stack -fstack-protector-all -fno-strict-aliasing -DOS_FREEBSD -DOS_FREEBSD -march=native PCAP_CFLAGS SECCFLAGS
------------------
output of gdb
------------------
# gdb /usr/local/bin/suricata suricata.core
GNU gdb (GDB) 8.2 [GDB v8.2 for FreeBSD]
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd11.2".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/bin/suricata...done.
[New LWP 100544]
[New LWP 100206]
[New LWP 100541]
[New LWP 100543]
[New LWP 100545]
[New LWP 100546]
[New LWP 100547]
[New LWP 100548]
[New LWP 100549]
[New LWP 100550]
Core was generated by `/usr/local/bin/suricata --netmap --pidfile /var/run/suricata.pid -c /usr/local/e'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000745b6c in SCACSearch (mpm_ctx=0x31162d0c330, mpm_thread_ctx=0x3117fdb7128, pmq=0x3117fdb7138, buf=0x31286776cef <error: Cannot access memory at address 0x31286776cef>, buflen=2636)
at util-mpm-ac.c:1048
1048 state = state_table_u16[state & 0x7FFF][u8_tolower(buf[i])];
[Current thread is 1 (LWP 100544)]
(gdb) set logging on
Copying output to gdb.txt.
(gdb) thread apply all bt
Thread 10 (LWP 100550):
#0 0x0000031115b29b5c in ?? () from /lib/libthr.so.3
#1 0x0000031115b1dbf1 in ?? () from /lib/libthr.so.3
#2 0x0000031115b27888 in ?? () from /lib/libthr.so.3
#3 0x00000000004e6744 in StatsMgmtThread (arg=0x3115c362f00) at counters.c:392
#4 0x0000031115b1bc36 in ?? () from /lib/libthr.so.3
#5 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x6bc326e7a000
Thread 9 (LWP 100549):
#0 0x0000031115b29b5c in ?? () from /lib/libthr.so.3
#1 0x0000031115b1dbf1 in ?? () from /lib/libthr.so.3
#2 0x0000031115b27888 in ?? () from /lib/libthr.so.3
#3 0x00000000004e5f71 in StatsWakeupThread (arg=0x3115c362e00) at counters.c:467
#4 0x0000031115b1bc36 in ?? () from /lib/libthr.so.3
#5 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x6bc32707b000
Thread 8 (LWP 100548):
#0 0x0000031115b29b5c in ?? () from /lib/libthr.so.3
#1 0x0000031115b1dbf1 in ?? () from /lib/libthr.so.3
#2 0x0000031115b27888 in ?? () from /lib/libthr.so.3
--Type <RET> for more, q to quit, c to continue without paging--c
#3 0x00000000006001c5 in FlowRecycler (th_v=0x3115c362d00, thread_data=0x3118200d000) at flow-manager.c:940
#4 0x00000000006f3cdd in TmThreadsManagement (td=0x3115c362d00) at tm-threads.c:719
#5 0x0000031115b1bc36 in ?? () from /lib/libthr.so.3
#6 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x6bc32727c000
Thread 7 (LWP 100547):
#0 0x0000031115b29b5c in ?? () from /lib/libthr.so.3
#1 0x0000031115b1dbf1 in ?? () from /lib/libthr.so.3
#2 0x0000031115b27888 in ?? () from /lib/libthr.so.3
#3 0x00000000005ff7d8 in FlowManager (th_v=0x3115c362c00, thread_data=0x3118180d000) at flow-manager.c:787
#4 0x00000000006f3cdd in TmThreadsManagement (td=0x3115c362c00) at tm-threads.c:719
#5 0x0000031115b1bc36 in ?? () from /lib/libthr.so.3
#6 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x6bc32747d000
Thread 6 (LWP 100546):
#0 0x000000000067c671 in OutputTxLog (tv=0x311810b7600, p=0x31180ffe2f0, thread_data=0x31180ffe2f0) at output-tx.c:131
#1 0x000000000063a228 in OutputLoggerLog (tv=0x3115c362b00, p=0x311810b7600, thread_data=0x31180ffe2f0) at output.c:921
#2 0x0000000000605998 in FlowWorker (tv=0x3115c362b00, p=0x311810b7600, data=0x311810da000, preq=0x3115ffba780, unused=0x0) at flow-worker.c:265
#3 0x00000000006ec067 in TmThreadsSlotVarRun (tv=0x3115c362b00, p=0x311810b7600, slot=0x3115ffba680) at tm-threads.c:145
#4 0x00000000006a9dd4 in TmThreadsSlotProcessPkt (tv=0x3115c362b00, s=0x3115ffba680, p=0x311810b7600) at ./tm-threads.h:147
#5 0x00000000006a9c68 in NetmapRingRead (ntv=0x311810bd000, ring_id=4) at source-netmap.c:816
#6 0x00000000006a767c in ReceiveNetmapLoop (tv=0x3115c362b00, data=0x311810bd000, slot=0x3115ffba5c0) at source-netmap.c:896
#7 0x00000000006f3739 in TmThreadsSlotPktAcqLoop (td=0x3115c362b00) at tm-threads.c:348
#8 0x0000031115b1bc36 in ?? () from /lib/libthr.so.3
#9 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x6bc32767e000
Thread 5 (LWP 100545):
#0 0x00000000006cf273 in TCPChecksum (shdr=0x3116c28c01a, pkt=0x3116c28c336, tlen=692, init=49835) at ./decode-tcp.h:197
#1 0x00000000006bd5b5 in StreamTcpValidateChecksum (p=0x311806b7600) at stream-tcp.c:4917
#2 0x00000000006bd456 in StreamTcp (tv=0x3115c362900, p=0x311806b7600, data=0x311806da050, pq=0x311806da020, postpq=0x0) at stream-tcp.c:5114
#3 0x0000000000605820 in FlowWorker (tv=0x3115c362900, p=0x311806b7600, data=0x311806da000, preq=0x3115ffba480, unused=0x0) at flow-worker.c:216
#4 0x00000000006ec067 in TmThreadsSlotVarRun (tv=0x3115c362900, p=0x311806b7600, slot=0x3115ffba380) at tm-threads.c:145
#5 0x00000000006a9dd4 in TmThreadsSlotProcessPkt (tv=0x3115c362900, s=0x3115ffba380, p=0x311806b7600) at ./tm-threads.h:147
#6 0x00000000006a9c68 in NetmapRingRead (ntv=0x311806bd000, ring_id=3) at source-netmap.c:816
#7 0x00000000006a767c in ReceiveNetmapLoop (tv=0x3115c362900, data=0x311806bd000, slot=0x3115ffba2c0) at source-netmap.c:896
#8 0x00000000006f3739 in TmThreadsSlotPktAcqLoop (td=0x3115c362900) at tm-threads.c:348
#9 0x0000031115b1bc36 in ?? () from /lib/libthr.so.3
#10 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x6bc32787f000
Thread 4 (LWP 100543):
#0 0x0000000000745b94 in SCACSearch (mpm_ctx=0x31162d0c330, mpm_thread_ctx=0x3117f3b7108, pmq=0x3117f3b7138, buf=0x3116c2bc042 <error: Cannot access memory at address 0x3116c2bc042>, buflen=1448) at util-mpm-ac.c:1049
#1 0x00000000005751de in PrefilterPktStream (det_ctx=0x3117f3b7000, p=0x3117f2b7600, pectx=0x31162d0c330) at detect-engine-payload.c:102
#2 0x000000000057adf3 in Prefilter (det_ctx=0x3117f3b7000, sgh=0x31162d7b7a0, p=0x3117f2b7600, flags=8 '\b') at detect-engine-prefilter.c:169
#3 0x000000000051626a in DetectRunPrefilterPkt (tv=0x3115c362700, de_ctx=0x3115b0c7000, det_ctx=0x3117f3b7000, p=0x3117f2b7600, scratch=0x3117f11df90) at detect.c:733
#4 0x00000000005158d2 in DetectRun (th_v=0x3115c362700, de_ctx=0x3115b0c7000, det_ctx=0x3117f3b7000, p=0x3117f2b7600) at detect.c:131
#5 0x0000000000515527 in DetectFlow (tv=0x3115c362700, de_ctx=0x3115b0c7000, det_ctx=0x3117f3b7000, p=0x3117f2b7600) at detect.c:1661
#6 0x0000000000515362 in Detect (tv=0x3115c362700, p=0x3117f2b7600, data=0x3117f3b7000, pq=0x0, postpq=0x0) at detect.c:1735
#7 0x000000000060597d in FlowWorker (tv=0x3115c362700, p=0x3117f2b7600, data=0x3117f2da000, preq=0x3115ffb9e80, unused=0x0) at flow-worker.c:260
#8 0x00000000006ec067 in TmThreadsSlotVarRun (tv=0x3115c362700, p=0x3117f2b7600, slot=0x3115ffb9d80) at tm-threads.c:145
#9 0x00000000006a9dd4 in TmThreadsSlotProcessPkt (tv=0x3115c362700, s=0x3115ffb9d80, p=0x3117f2b7600) at ./tm-threads.h:147
#10 0x00000000006a9c68 in NetmapRingRead (ntv=0x3117f2bd000, ring_id=1) at source-netmap.c:816
#11 0x00000000006a767c in ReceiveNetmapLoop (tv=0x3115c362700, data=0x3117f2bd000, slot=0x3115ffb9cc0) at source-netmap.c:896
#12 0x00000000006f3739 in TmThreadsSlotPktAcqLoop (td=0x3115c362700) at tm-threads.c:348
#13 0x0000031115b1bc36 in ?? () from /lib/libthr.so.3
#14 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x6bc327c81000
Thread 3 (LWP 100541):
#0 0x000000000074a3c1 in __getCurrentRuneLocale () at /usr/include/runetype.h:94
#1 0x0000000000746375 in __sbtolower (_c=42) at /usr/include/_ctype.h:155
#2 0x0000000000745b7c in SCACSearch (mpm_ctx=0x31162d0c330, mpm_thread_ctx=0x31165db8108, pmq=0x31165db8138, buf=0x3116b328042 <error: Cannot access memory at address 0x3116b328042>, buflen=1448) at util-mpm-ac.c:1048
#3 0x00000000005751de in PrefilterPktStream (det_ctx=0x31165db8000, p=0x31165cb7600, pectx=0x31162d0c330) at detect-engine-payload.c:102
#4 0x000000000057adf3 in Prefilter (det_ctx=0x31165db8000, sgh=0x31162d79950, p=0x31165cb7600, flags=8 '\b') at detect-engine-prefilter.c:169
#5 0x000000000051626a in DetectRunPrefilterPkt (tv=0x3115c362600, de_ctx=0x3115b0c7000, det_ctx=0x31165db8000, p=0x31165cb7600, scratch=0x311667fcf90) at detect.c:733
#6 0x00000000005158d2 in DetectRun (th_v=0x3115c362600, de_ctx=0x3115b0c7000, det_ctx=0x31165db8000, p=0x31165cb7600) at detect.c:131
#7 0x0000000000515527 in DetectFlow (tv=0x3115c362600, de_ctx=0x3115b0c7000, det_ctx=0x31165db8000, p=0x31165cb7600) at detect.c:1661
#8 0x0000000000515362 in Detect (tv=0x3115c362600, p=0x31165cb7600, data=0x31165db8000, pq=0x0, postpq=0x0) at detect.c:1735
#9 0x000000000060597d in FlowWorker (tv=0x3115c362600, p=0x31165cb7600, data=0x31165cdb000, preq=0x3115ffb9b80, unused=0x0) at flow-worker.c:260
#10 0x00000000006ec067 in TmThreadsSlotVarRun (tv=0x3115c362600, p=0x31165cb7600, slot=0x3115ffb9a80) at tm-threads.c:145
#11 0x00000000006a9dd4 in TmThreadsSlotProcessPkt (tv=0x3115c362600, s=0x3115ffb9a80, p=0x31165cb7600) at ./tm-threads.h:147
#12 0x00000000006a9c68 in NetmapRingRead (ntv=0x31165cbd000, ring_id=0) at source-netmap.c:816
#13 0x00000000006a767c in ReceiveNetmapLoop (tv=0x3115c362600, data=0x31165cbd000, slot=0x3115ffb99c0) at source-netmap.c:896
#14 0x00000000006f3739 in TmThreadsSlotPktAcqLoop (td=0x3115c362600) at tm-threads.c:348
#15 0x0000031115b1bc36 in ?? () from /lib/libthr.so.3
#16 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x6bc327e82000
Thread 2 (LWP 100206):
#0 0x0000031116fa85da in _nanosleep () from /lib/libc.so.7
#1 0x0000031115b1e63c in ?? () from /lib/libthr.so.3
#2 0x0000031116ff98c6 in usleep () from /lib/libc.so.7
#3 0x00000000006e2f49 in SuricataMainLoop (suri=0xc78cd8 <suricata>) at suricata.c:2883
#4 0x00000000006db8b4 in main (argc=6, argv=0x6bc367e82948) at suricata.c:3026
Thread 1 (LWP 100544):
#0 0x0000000000745b6c in SCACSearch (mpm_ctx=0x31162d0c330, mpm_thread_ctx=0x3117fdb7128, pmq=0x3117fdb7138, buf=0x31286776cef <error: Cannot access memory at address 0x31286776cef>, buflen=2636) at util-mpm-ac.c:1048
#1 0x0000000000575c2a in StreamMpmFunc (cb_data=0x3117f7fcf78, data=0x31286776cef <error: Cannot access memory at address 0x31286776cef>, data_len=2636) at detect-engine-payload.c:64
#2 0x00000000006d66eb in StreamReassembleRawInline (ssn=0x3118622cc00, p=0x3117fcb7600, Callback=0x575b90 <StreamMpmFunc>, cb_data=0x3117f7fcf78, progress_out=0x3117fdb7050) at stream-tcp-reassemble.c:1470
#3 0x00000000006d5f2c in StreamReassembleRaw (ssn=0x3118622cc00, p=0x3117fcb7600, Callback=0x575b90 <StreamMpmFunc>, cb_data=0x3117f7fcf78, progress_out=0x3117fdb7050, respect_inspect_depth=false) at stream-tcp-reassemble.c:1660
#4 0x000000000057512a in PrefilterPktStream (det_ctx=0x3117fdb7000, p=0x3117fcb7600, pectx=0x31162d0c330) at detect-engine-payload.c:83
#5 0x000000000057adf3 in Prefilter (det_ctx=0x3117fdb7000, sgh=0x31162d78c30, p=0x3117fcb7600, flags=136 '\210') at detect-engine-prefilter.c:169
#6 0x000000000051626a in DetectRunPrefilterPkt (tv=0x3115c362800, de_ctx=0x3115b0c7000, det_ctx=0x3117fdb7000, p=0x3117fcb7600, scratch=0x3117f7fcf90) at detect.c:733
#7 0x00000000005158d2 in DetectRun (th_v=0x3115c362800, de_ctx=0x3115b0c7000, det_ctx=0x3117fdb7000, p=0x3117fcb7600) at detect.c:131
#8 0x0000000000515527 in DetectFlow (tv=0x3115c362800, de_ctx=0x3115b0c7000, det_ctx=0x3117fdb7000, p=0x3117fcb7600) at detect.c:1661
#9 0x0000000000515362 in Detect (tv=0x3115c362800, p=0x3117fcb7600, data=0x3117fdb7000, pq=0x0, postpq=0x0) at detect.c:1735
#10 0x000000000060597d in FlowWorker (tv=0x3115c362800, p=0x3117fcb7600, data=0x3117fcda000, preq=0x3115ffba180, unused=0x0) at flow-worker.c:260
#11 0x00000000006ec067 in TmThreadsSlotVarRun (tv=0x3115c362800, p=0x3117fcb7600, slot=0x3115ffba080) at tm-threads.c:145
#12 0x00000000006a9dd4 in TmThreadsSlotProcessPkt (tv=0x3115c362800, s=0x3115ffba080, p=0x3117fcb7600) at ./tm-threads.h:147
#13 0x00000000006a9c68 in NetmapRingRead (ntv=0x3117fcbd000, ring_id=2) at source-netmap.c:816
#14 0x00000000006a767c in ReceiveNetmapLoop (tv=0x3115c362800, data=0x3117fcbd000, slot=0x3115ffb9fc0) at source-netmap.c:896
#15 0x00000000006f3739 in TmThreadsSlotPktAcqLoop (td=0x3115c362800) at tm-threads.c:348
#16 0x0000031115b1bc36 in ?? () from /lib/libthr.so.3
#17 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x6bc327a80000
Actions