Bug #3681: Rule reload causes segfault - Suricata - Open Information Security Foundation

Actions

Copy link

Bug #3681

closed

Rule reload causes segfault

Added by Sascha Steinbiss almost 4 years ago. Updated almost 4 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Jeff Lucovsky

Target version:

TBD

Affected Versions:

6.0.0beta1

Effort:

Difficulty:

Label:

Description

I noticed that when testing the 6.0.0 code on our sensors, rule reloads triggered via suricatasc fail with a segfault. Here's a gdb session:

GNU gdb (Debian 8.2.1-2+b3) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./src/.libs/suricata...done.
(gdb) r --af-packet -c /etc/suricata/suricata-dcso.yaml --pidfile /run/suricata.pid
Starting program: /home/dcsoadm/suri-upstream/src/.libs/suricata --af-packet -c /etc/suricata/suricata-dcso.yaml --pidfile /run/suricata.pid
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[38495] 27/4/2020 -- 15:28:32 - (conf-yaml-loader.c:267) <Info> (ConfYamlParse) -- Including configuration file customer-vars.yaml.
[38495] 27/4/2020 -- 15:28:32 - (conf-yaml-loader.c:267) <Info> (ConfYamlParse) -- Including configuration file dcso-rules.yaml.
[38495] 27/4/2020 -- 15:28:32 - (conf-yaml-loader.c:267) <Info> (ConfYamlParse) -- Including configuration file dcso-logging.yaml.
[38495] 27/4/2020 -- 15:28:32 - (conf-yaml-loader.c:267) <Info> (ConfYamlParse) -- Including configuration file dcso-applayer.yaml.
[38495] 27/4/2020 -- 15:28:32 - (conf-yaml-loader.c:267) <Info> (ConfYamlParse) -- Including configuration file dcso-advanced.yaml.
[38495] 27/4/2020 -- 15:28:32 - (conf-yaml-loader.c:267) <Info> (ConfYamlParse) -- Including configuration file dcso-affinity.yaml.
[38495] 27/4/2020 -- 15:28:32 - (conf-yaml-loader.c:267) <Info> (ConfYamlParse) -- Including configuration file dcso-interfaces.yaml.
[38495] 27/4/2020 -- 15:28:32 - (suricata.c:1057) <Notice> (LogVersion) -- This is Suricata version 6.0.0-dev (eef776087 2020-04-27) running in SYSTEM mode
[38495] 27/4/2020 -- 15:28:32 - (util-logopenfile.c:81) <Warning> (SCLogOpenUnixSocketFp) -- [ERRCODE: SC_ERR_SOCKET(200)] - Error connecting to socket "/tmp/files.sock": No such file or directory (will keep trying)
[38495] 27/4/2020 -- 15:28:32 - (output-tx.c:77) <Notice> (OutputRegisterTxLogger) -- JsonRdpLog logger not enabled: protocol rdp is disabled

[38495] 27/4/2020 -- 15:30:02 - (runmode-af-packet.c:585) <Error> (ParseAFPConfig) -- [ERRCODE: SC_ERR_INVALID_VALUE(130)] - Block-size must be a multiple of pagesize.
[New Thread 0x7ffff240e700 (LWP 39053)]
[New Thread 0x7ffff1989700 (LWP 39054)]
[New Thread 0x7ffff1108700 (LWP 39055)]
[New Thread 0x7ffff0887700 (LWP 39056)]
[New Thread 0x7fffb3fff700 (LWP 39057)]
[New Thread 0x7fffb377e700 (LWP 39058)]
[New Thread 0x7fffb2efd700 (LWP 39059)]
[New Thread 0x7fffb267c700 (LWP 39060)]
[New Thread 0x7fffb1dfb700 (LWP 39061)]
[38495] 27/4/2020 -- 15:30:03 - (runmode-af-packet.c:585) <Error> (ParseAFPConfig) -- [ERRCODE: SC_ERR_INVALID_VALUE(130)] - Block-size must be a multiple of pagesize.
[New Thread 0x7fffb157a700 (LWP 39062)]
[New Thread 0x7fffb0cf9700 (LWP 39063)]
[New Thread 0x7fff3ffff700 (LWP 39064)]
[New Thread 0x7fff3f7fe700 (LWP 39065)]
[New Thread 0x7fff3effd700 (LWP 39066)]
[New Thread 0x7fff3e7fc700 (LWP 39067)]
[New Thread 0x7fff3dffb700 (LWP 39068)]
[New Thread 0x7fff3d7fa700 (LWP 39069)]
[New Thread 0x7fff3cff9700 (LWP 39070)]
[New Thread 0x7ffecbfff700 (LWP 39071)]
[New Thread 0x7ffecb7fe700 (LWP 39072)]
[New Thread 0x7ffecaffd700 (LWP 39073)]
[New Thread 0x7ffeca7fc700 (LWP 39074)]
[New Thread 0x7ffec9ffb700 (LWP 39075)]
[New Thread 0x7ffec97fa700 (LWP 39076)]
[New Thread 0x7ffec8ff9700 (LWP 39077)]
[38495] 27/4/2020 -- 15:30:06 - (tm-threads.c:1888) <Notice> (TmThreadWaitOnThreadInit) -- all 18 packet processing threads, 6 management threads initialized, engine started.
[39066] 27/4/2020 -- 15:30:08 - (util-log-redis.c:252) <Notice> (SCLogRedisWriteAsync) -- Trying to connect to Redis
[39066] 27/4/2020 -- 15:30:08 - (util-log-redis.c:134) <Notice> (SCRedisAsyncEchoCommandCallback) -- Connected to Redis.
[38495] 27/4/2020 -- 15:30:26 - (detect-engine.c:4008) <Notice> (DetectEngineReload) -- rule reload starting
Thread 17 "W#07-enp175s0f1" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff3dffb700 (LWP 39068)]
0x00007ffd25c1cbb2 in ?? ()
(gdb) bt
#0  0x00007ffd25c1cbb2 in ?? ()
#1  0x00007fff3dffa080 in ?? ()
#2  0x000055556dcc1600 in ?? ()
#3  0x00007ffd600be664 in ?? ()
#4  0x0000000000000000 in ?? ()
(gdb) info threads
  Id   Target Id                                           Frame 
  1    Thread 0x7ffff4d59d00 (LWP 38495) "Suricata-Main"   0x00007ffff5798720 in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
  2    Thread 0x7ffff240e700 (LWP 39053) "W#01-enp175s0f0" 0x00007ffff57c0819 in poll () from /lib/x86_64-linux-gnu/libc.so.6
  3    Thread 0x7ffff1989700 (LWP 39054) "W#02-enp175s0f0" 0x00007ffff57c0819 in poll () from /lib/x86_64-linux-gnu/libc.so.6
  4    Thread 0x7ffff1108700 (LWP 39055) "W#03-enp175s0f0" 0x00007ffff57c0819 in poll () from /lib/x86_64-linux-gnu/libc.so.6
  5    Thread 0x7ffff0887700 (LWP 39056) "W#04-enp175s0f0" 0x00007ffff57c0819 in poll () from /lib/x86_64-linux-gnu/libc.so.6
  6    Thread 0x7fffb3fff700 (LWP 39057) "W#05-enp175s0f0" 0x00007ffff57c0819 in poll () from /lib/x86_64-linux-gnu/libc.so.6
  7    Thread 0x7fffb377e700 (LWP 39058) "W#06-enp175s0f0" 0x00007ffff57c0819 in poll () from /lib/x86_64-linux-gnu/libc.so.6
  8    Thread 0x7fffb2efd700 (LWP 39059) "W#07-enp175s0f0" 0x00007ffff57c0819 in poll () from /lib/x86_64-linux-gnu/libc.so.6
  9    Thread 0x7fffb267c700 (LWP 39060) "W#08-enp175s0f0" 0x00007ffff57c0819 in poll () from /lib/x86_64-linux-gnu/libc.so.6
  10   Thread 0x7fffb1dfb700 (LWP 39061) "W#09-enp175s0f0" 0x00007ffff57c0819 in poll () from /lib/x86_64-linux-gnu/libc.so.6
  11   Thread 0x7fffb157a700 (LWP 39062) "W#01-enp175s0f1" 0x00007ffff6185f33 in ?? () from /usr/lib/x86_64-linux-gnu/libhs.so.5
  12   Thread 0x7fffb0cf9700 (LWP 39063) "W#02-enp175s0f1" lj_alloc_malloc (msp=0x44040010, nsize=<optimized out>) at lj_alloc.c:1348
  13   Thread 0x7fff3ffff700 (LWP 39064) "W#03-enp175s0f1" lj_alloc_malloc (msp=0x44000010, nsize=<optimized out>) at lj_alloc.c:1348
  14   Thread 0x7fff3f7fe700 (LWP 39065) "W#04-enp175s0f1" gc_onestep (L=L@entry=0x43fa0378) at lj_gc.c:616
  15   Thread 0x7fff3effd700 (LWP 39066) "W#05-enp175s0f1" SigMatchSignaturesGetSgh (de_ctx=de_ctx@entry=0x55556cdfb4d0, p=p@entry=0x7fff11db3710) at detect.c:218
  16   Thread 0x7fff3e7fc700 (LWP 39067) "W#06-enp175s0f1" 0x00007ffff6240704 in ?? () from /usr/lib/x86_64-linux-gnu/libhs.so.5
* 17   Thread 0x7fff3dffb700 (LWP 39068) "W#07-enp175s0f1" 0x00007ffd25c1cbb2 in ?? ()
  18   Thread 0x7fff3d7fa700 (LWP 39069) "W#08-enp175s0f1" 0x00007ffff575638b in malloc () from /lib/x86_64-linux-gnu/libc.so.6
  19   Thread 0x7fff3cff9700 (LWP 39070) "W#09-enp175s0f1" 0x00007ffff717a408 in lj_BC_TSETS () at buildvm_x86.dasc:571
  20   Thread 0x7ffecbfff700 (LWP 39071) "FM#01"           0x00007ffff688035b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  21   Thread 0x7ffecb7fe700 (LWP 39072) "FM#02"           0x00007ffff688035b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  22   Thread 0x7ffecaffd700 (LWP 39073) "FR#01"           0x00007ffff688035b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  23   Thread 0x7ffeca7fc700 (LWP 39074) "FR#02"           0x00007ffff688035b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  24   Thread 0x7ffec9ffb700 (LWP 39075) "CW"              0x00007ffff688035b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  25   Thread 0x7ffec97fa700 (LWP 39076) "CS"              0x00007ffff688035b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  26   Thread 0x7ffec8ff9700 (LWP 39077) "US"              0x00007ffff5798720 in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6

Looks like one thread ends up in weird memory that does not seem to map to actual code.

Compiled with:

./configure --build=x86_64-linux-gnu --prefix=/usr --includedir=\${prefix}/include --mandir=\${prefix}/share/man --infodir=\${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-silent-rules --libdir=\${prefix}/lib/x86_64-linux-gnu --runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking --enable-af-packet --enable-nfqueue --enable-prelude --enable-nflog --enable-gccprotect --disable-gccmarch-native --with-libnss-includes=/usr/include/nss --with-libnss-libraries=/usr/lib/x86_64-linux-gnu --with-libnspr-includes=/usr/include/nspr --with-libnspr-libraries=/usr/lib/x86_64-linux-gnu --with-libevent-includes=/usr/include --with-libevent-libraries=/usr/lib/x86_64-linux-gnu --disable-coccinelle --enable-geoip --enable-hiredis --enable-luajit --enable-rust

Result configuration:

Suricata Configuration:
  AF_PACKET support:                       yes
  eBPF support:                            no
  XDP support:                             no
  PF_RING support:                         no
  NFQueue support:                         yes
  NFLOG support:                           yes
  IPFW support:                            no
  Netmap support:                          no 
  DAG enabled:                             no
  Napatech enabled:                        no
  WinDivert enabled:                       no

  Unix socket enabled:                     yes
  Detection enabled:                       yes

  Libmagic support:                        yes
  libnss support:                          yes
  libnspr support:                         yes
  libjansson support:                      yes
  hiredis support:                         yes
  hiredis async with libevent:             yes
  Prelude support:                         yes
  PCRE jit:                                yes
  LUA support:                             yes, through luajit
  libluajit:                               yes
  GeoIP2 support:                          yes
  Non-bundled htp:                         no
  Old barnyard2 support:                   
  Hyperscan support:                       yes
  Libnet support:                          yes
  liblz4 support:                          yes

  Rust support:                            yes
  Rust strict mode:                        no
  Rust compiler path:                      /usr/bin/rustc
  Rust compiler version:                   rustc 1.34.2
  Cargo path:                              /usr/bin/cargo
  Cargo version:                           cargo 1.34.0
  Cargo vendor:                            no

  Python support:                          yes
  Python path:                             /usr/bin/python3
  Python distutils                         yes
  Python yaml                              yes
  Install suricatactl:                     yes
  Install suricatasc:                      yes
  Install suricata-update:                 not bundled

  Profiling enabled:                       no
  Profiling locks enabled:                 no

Development settings:
  Coccinelle / spatch:                     no
  Unit tests enabled:                      no
  Debug output enabled:                    no
  Debug validation enabled:                no

Generic build parameters:
  Installation prefix:                     /usr
  Configuration directory:                 /etc/suricata/
  Log directory:                           /var/log/suricata/

  --prefix                                 /usr
  --sysconfdir                             /etc
  --localstatedir                          /var
  --datarootdir                            /usr/share

  Host:                                    x86_64-pc-linux-gnu
  Compiler:                                gcc (exec name) / c++ (real)
  GCC Protect enabled:                     yes
  GCC march native enabled:                no
  GCC Profile enabled:                     no
  Position Independent Executable enabled: no
  CFLAGS                                   -g -O2 -std=c11 -I${srcdir}/../rust/gen
  PCAP_CFLAGS                               -I/usr/include
  SECCFLAGS                                -fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security

Suricata 5.0.2 does not show this issue. I have used git-bisect between the 5.0.1 tag and the current master (eef7760870d99beca75cf96262f4721563198a42) to narrow down the problem and it seeems to be one of the following commits:

aa67a0a236d6544301caac4ba8c74d2951926b92
abe0cdc4adc872f346a94c554883570783917034
d19429f7e54f3d8e1d1c0c11470c1cabeca3f47a
4b0085b03ce85f9b0f09d7e44a96774388d0b09b

I couldn't trace it any deeper because most of these didn't build for me with errors like:

detect-byte-extract.c: In function ‘DetectByteExtractRegister’:
detect-byte-extract.c:109:42: warning: passing argument 2 of ‘DetectSetupParseRegexes’ from incompatible pointer type [-Wincompatible-pointer-types]
     DetectSetupParseRegexes(PARSE_REGEX, &parse_regex, &parse_regex_study);
                                          ^~~~~~~~~~~~
In file included from detect-byte-extract.c:29:
detect-parse.h:92:71: note: expected ‘DetectParseRegex *’ {aka ‘struct DetectParseRegex_ *’} but argument is of type ‘pcre **’ {aka ‘struct real_pcre **’}
 void DetectSetupParseRegexes(const char *parse_str, DetectParseRegex *parse_regex);
                                                     ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~
detect-byte-extract.c:109:5: error: too many arguments to function ‘DetectSetupParseRegexes’
     DetectSetupParseRegexes(PARSE_REGEX, &parse_regex, &parse_regex_study);
     ^~~~~~~~~~~~~~~~~~~~~~~
In file included from detect-byte-extract.c:29:
detect-parse.h:92:6: note: declared here
 void DetectSetupParseRegexes(const char *parse_str, DetectParseRegex *parse_regex);
      ^~~~~~~~~~~~~~~~~~~~~~~
detect-byte-extract.c: In function ‘DetectByteExtractParse’:
detect-byte-extract.c:214: warning: "MAX_SUBSTRINGS" redefined
 #define MAX_SUBSTRINGS 100

In file included from detect-byte-extract.c:29:
detect-parse.h:110: note: this is the location of the previous definition
 #define MAX_SUBSTRINGS 30

gcc -DHAVE_CONFIG_H -I. -I..   -I./../libhtp/  -I/usr/include/hs  -I/usr/include/nspr -I/usr/include/nspr -I/usr/include/nss -I/usr/include/nspr -I/usr/include/nss -I/usr/include/luajit-2.1 -I/usr/include  -Wextra -Werror-implicit-function-declaration  -fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I/usr/include -DLOCAL_STATE_DIR=\"/var\" -std=gnu99 -Wall -Wno-unused-parameter -Wmissing-prototypes -Wmissing-declarations -Wstrict-prototypes -Wwrite-strings -Wbad-function-cast -Wformat-security -Wno-format-nonliteral -Wmissing-format-attribute -funsigned-char  -g -O2 -I./../rust/gen -c -o detect-detection-filter.o detect-detection-filter.c
detect-bytejump.c: In function ‘DetectBytejumpRegister’:
detect-bytejump.c:80:42: warning: passing argument 2 of ‘DetectSetupParseRegexes’ from incompatible pointer type [-Wincompatible-pointer-types]
     DetectSetupParseRegexes(PARSE_REGEX, &parse_regex, &parse_regex_study);
                                          ^~~~~~~~~~~~
In file included from detect-bytejump.c:30:
detect-parse.h:92:71: note: expected ‘DetectParseRegex *’ {aka ‘struct DetectParseRegex_ *’} but argument is of type ‘pcre **’ {aka ‘struct real_pcre **’}
 void DetectSetupParseRegexes(const char *parse_str, DetectParseRegex *parse_regex);
                                                     ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~
detect-bytejump.c:80:5: error: too many arguments to function ‘DetectSetupParseRegexes’
     DetectSetupParseRegexes(PARSE_REGEX, &parse_regex, &parse_regex_study);
     ^~~~~~~~~~~~~~~~~~~~~~~
In file included from detect-bytejump.c:30:
detect-parse.h:92:6: note: declared here
 void DetectSetupParseRegexes(const char *parse_str, DetectParseRegex *parse_regex);
      ^~~~~~~~~~~~~~~~~~~~~~~
detect-base64-decode.c: In function ‘DetectBase64DecodeRegister’:
detect-base64-decode.c:53:45: warning: passing argument 2 of ‘DetectSetupParseRegexes’ from incompatible pointer type [-Wincompatible-pointer-types]
     DetectSetupParseRegexes(decode_pattern, &decode_pcre, &decode_pcre_study);
                                             ^~~~~~~~~~~~
In file included from detect-base64-decode.c:20:
detect-parse.h:92:71: note: expected ‘DetectParseRegex *’ {aka ‘struct DetectParseRegex_ *’} but argument is of type ‘pcre **’ {aka ‘struct real_pcre **’}
 void DetectSetupParseRegexes(const char *parse_str, DetectParseRegex *parse_regex);
                                                     ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~
detect-base64-decode.c:53:5: error: too many arguments to function ‘DetectSetupParseRegexes’
     DetectSetupParseRegexes(decode_pattern, &decode_pcre, &decode_pcre_study);
     ^~~~~~~~~~~~~~~~~~~~~~~
In file included from detect-base64-decode.c:20:
detect-parse.h:92:6: note: declared here
 void DetectSetupParseRegexes(const char *parse_str, DetectParseRegex *parse_regex);
      ^~~~~~~~~~~~~~~~~~~~~~~
gcc -DHAVE_CONFIG_H -I. -I..   -I./../libhtp/  -I/usr/include/hs  -I/usr/include/nspr -I/usr/include/nspr -I/usr/include/nss -I/usr/include/nspr -I/usr/include/nss -I/usr/include/luajit-2.1 -I/usr/include  -Wextra -Werror-implicit-function-declaration  -fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -I/usr/include -DLOCAL_STATE_DIR=\"/var\" -std=gnu99 -Wall -Wno-unused-parameter -Wmissing-prototypes -Wmissing-declarations -Wstrict-prototypes -Wwrite-strings -Wbad-function-cast -Wformat-security -Wno-format-nonliteral -Wmissing-format-attribute -funsigned-char  -g -O2 -I./../rust/gen -c -o detect-distance.o detect-distance.c
detect-bytetest.c: In function ‘DetectBytetestRegister’:
detect-bytetest.c:81:42: warning: passing argument 2 of ‘DetectSetupParseRegexes’ from incompatible pointer type [-Wincompatible-pointer-types]
     DetectSetupParseRegexes(PARSE_REGEX, &parse_regex, &parse_regex_study);
                                          ^~~~~~~~~~~~
In file included from detect-bytetest.c:31:
detect-parse.h:92:71: note: expected ‘DetectParseRegex *’ {aka ‘struct DetectParseRegex_ *’} but argument is of type ‘pcre **’ {aka ‘struct real_pcre **’}
 void DetectSetupParseRegexes(const char *parse_str, DetectParseRegex *parse_regex);
                                                     ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~
detect-bytetest.c:81:5: error: too many arguments to function ‘DetectSetupParseRegexes’
     DetectSetupParseRegexes(PARSE_REGEX, &parse_regex, &parse_regex_study);
     ^~~~~~~~~~~~~~~~~~~~~~~
In file included from detect-bytetest.c:31:
detect-parse.h:92:6: note: declared here
 void DetectSetupParseRegexes(const char *parse_str, DetectParseRegex *parse_regex);
      ^~~~~~~~~~~~~~~~~~~~~~~
make[2]: *** [Makefile:2170: detect-base64-decode.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: *** [Makefile:2170: detect-byte-extract.o] Error 1
make[2]: *** [Makefile:2170: detect-bytejump.o] Error 1
make[2]: *** [Makefile:2170: detect-bytetest.o] Error 1
detect-datarep.c: In function ‘DetectDatarepRegister’:
detect-datarep.c:59:42: warning: passing argument 2 of ‘DetectSetupParseRegexes’ from incompatible pointer type [-Wincompatible-pointer-types]
     DetectSetupParseRegexes(PARSE_REGEX, &parse_regex, &parse_regex_study);
                                          ^~~~~~~~~~~~
In file included from detect-datarep.c:33:
detect-parse.h:92:71: note: expected ‘DetectParseRegex *’ {aka ‘struct DetectParseRegex_ *’} but argument is of type ‘pcre **’ {aka ‘struct real_pcre **’}
 void DetectSetupParseRegexes(const char *parse_str, DetectParseRegex *parse_regex);
                                                     ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~
detect-datarep.c:59:5: error: too many arguments to function ‘DetectSetupParseRegexes’
     DetectSetupParseRegexes(PARSE_REGEX, &parse_regex, &parse_regex_study);
     ^~~~~~~~~~~~~~~~~~~~~~~
In file included from detect-datarep.c:33:
detect-parse.h:92:6: note: declared here
 void DetectSetupParseRegexes(const char *parse_str, DetectParseRegex *parse_regex);
      ^~~~~~~~~~~~~~~~~~~~~~~
detect-dce-iface.c: In function ‘DetectDceIfaceRegister’:
detect-dce-iface.c:83:42: warning: passing argument 2 of ‘DetectSetupParseRegexes’ from incompatible pointer type [-Wincompatible-pointer-types]
     DetectSetupParseRegexes(PARSE_REGEX, &parse_regex, &parse_regex_study);
                                          ^~~~~~~~~~~~
In file included from detect-dce-iface.c:29:
detect-parse.h:92:71: note: expected ‘DetectParseRegex *’ {aka ‘struct DetectParseRegex_ *’} but argument is of type ‘pcre **’ {aka ‘struct real_pcre **’}
 void DetectSetupParseRegexes(const char *parse_str, DetectParseRegex *parse_regex);
                                                     ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~
detect-dce-iface.c:83:5: error: too many arguments to function ‘DetectSetupParseRegexes’
     DetectSetupParseRegexes(PARSE_REGEX, &parse_regex, &parse_regex_study);
     ^~~~~~~~~~~~~~~~~~~~~~~
In file included from detect-dce-iface.c:29:
detect-parse.h:92:6: note: declared here
 void DetectSetupParseRegexes(const char *parse_str, DetectParseRegex *parse_regex);
      ^~~~~~~~~~~~~~~~~~~~~~~
make[2]: *** [Makefile:2170: detect-datarep.o] Error 1
detect-dataset.c: In function ‘DetectDatasetRegister’:
detect-dataset.c:58:42: warning: passing argument 2 of ‘DetectSetupParseRegexes’ from incompatible pointer type [-Wincompatible-pointer-types]
     DetectSetupParseRegexes(PARSE_REGEX, &parse_regex, &parse_regex_study);
                                          ^~~~~~~~~~~~
In file included from detect-dataset.c:33:
detect-parse.h:92:71: note: expected ‘DetectParseRegex *’ {aka ‘struct DetectParseRegex_ *’} but argument is of type ‘pcre **’ {aka ‘struct real_pcre **’}
 void DetectSetupParseRegexes(const char *parse_str, DetectParseRegex *parse_regex);
                                                     ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~
detect-dataset.c:58:5: error: too many arguments to function ‘DetectSetupParseRegexes’
     DetectSetupParseRegexes(PARSE_REGEX, &parse_regex, &parse_regex_study);
     ^~~~~~~~~~~~~~~~~~~~~~~
In file included from detect-dataset.c:33:
detect-parse.h:92:6: note: declared here
 void DetectSetupParseRegexes(const char *parse_str, DetectParseRegex *parse_regex);
      ^~~~~~~~~~~~~~~~~~~~~~~
detect-classtype.c: In function ‘DetectClasstypeRegister’:
detect-classtype.c:58:42: warning: passing argument 2 of ‘DetectSetupParseRegexes’ from incompatible pointer type [-Wincompatible-pointer-types]
     DetectSetupParseRegexes(PARSE_REGEX, &regex, &regex_study);
                                          ^~~~~~
In file included from detect-classtype.c:31:
detect-parse.h:92:71: note: expected ‘DetectParseRegex *’ {aka ‘struct DetectParseRegex_ *’} but argument is of type ‘pcre **’ {aka ‘struct real_pcre **’}
 void DetectSetupParseRegexes(const char *parse_str, DetectParseRegex *parse_regex);
                                                     ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~
detect-classtype.c:58:5: error: too many arguments to function ‘DetectSetupParseRegexes’
     DetectSetupParseRegexes(PARSE_REGEX, &regex, &regex_study);
     ^~~~~~~~~~~~~~~~~~~~~~~
In file included from detect-classtype.c:31:
detect-parse.h:92:6: note: declared here
 void DetectSetupParseRegexes(const char *parse_str, DetectParseRegex *parse_regex);
      ^~~~~~~~~~~~~~~~~~~~~~~
detect-dce-opnum.c: In function ‘DetectDceOpnumRegister’:
detect-dce-opnum.c:78:42: warning: passing argument 2 of ‘DetectSetupParseRegexes’ from incompatible pointer type [-Wincompatible-pointer-types]
     DetectSetupParseRegexes(PARSE_REGEX, &parse_regex, &parse_regex_study);
                                          ^~~~~~~~~~~~
In file included from detect-dce-opnum.c:29:
detect-parse.h:92:71: note: expected ‘DetectParseRegex *’ {aka ‘struct DetectParseRegex_ *’} but argument is of type ‘pcre **’ {aka ‘struct real_pcre **’}
 void DetectSetupParseRegexes(const char *parse_str, DetectParseRegex *parse_regex);
                                                     ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~
detect-dce-opnum.c:78:5: error: too many arguments to function ‘DetectSetupParseRegexes’
     DetectSetupParseRegexes(PARSE_REGEX, &parse_regex, &parse_regex_study);
     ^~~~~~~~~~~~~~~~~~~~~~~
In file included from detect-dce-opnum.c:29:
detect-parse.h:92:6: note: declared here
 void DetectSetupParseRegexes(const char *parse_str, DetectParseRegex *parse_regex);
      ^~~~~~~~~~~~~~~~~~~~~~~
detect-detection-filter.c: In function ‘DetectDetectionFilterRegister’:
detect-detection-filter.c:75:42: warning: passing argument 2 of ‘DetectSetupParseRegexes’ from incompatible pointer type [-Wincompatible-pointer-types]
     DetectSetupParseRegexes(PARSE_REGEX, &parse_regex, &parse_regex_study);
                                          ^~~~~~~~~~~~
In file included from detect-detection-filter.c:36:
detect-parse.h:92:71: note: expected ‘DetectParseRegex *’ {aka ‘struct DetectParseRegex_ *’} but argument is of type ‘pcre **’ {aka ‘struct real_pcre **’}
 void DetectSetupParseRegexes(const char *parse_str, DetectParseRegex *parse_regex);
                                                     ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~
detect-detection-filter.c:75:5: error: too many arguments to function ‘DetectSetupParseRegexes’
     DetectSetupParseRegexes(PARSE_REGEX, &parse_regex, &parse_regex_study);
     ^~~~~~~~~~~~~~~~~~~~~~~
In file included from detect-detection-filter.c:36:
detect-parse.h:92:6: note: declared here
 void DetectSetupParseRegexes(const char *parse_str, DetectParseRegex *parse_regex);
      ^~~~~~~~~~~~~~~~~~~~~~~
make[2]: *** [Makefile:2170: detect-dataset.o] Error 1
make[2]: *** [Makefile:2170: detect-detection-filter.o] Error 1
make[2]: *** [Makefile:2170: detect-dce-opnum.o] Error 1
make[2]: *** [Makefile:2170: detect-dce-iface.o] Error 1
make[2]: *** [Makefile:2170: detect-classtype.o] Error 1

Actions

Copy link

Updated by Andreas Herz almost 4 years ago

Assignee set to Jeff Lucovsky
Target version set to TBD

@Jeff Lucovsky can you look into those 4 commits from you if you see anything that could be the reason for that segfault?

Actions

Copy link

Updated by Jason Ish almost 4 years ago

This is no better than a "work for me" post, but initial attempts to replicate in my development environment (Fedora 31) have been unsuccessful.

Actions

Copy link

Updated by Sascha Steinbiss almost 4 years ago

I can look deeper into how different configuration options might influence this issue later today.

Actions

Copy link

Updated by Victor Julien almost 4 years ago

Are you able to build suri with asan enabled?

The bisect failure shouldn't happen anymore in the future. We added a CI check to build each commit.

Actions

Copy link

Updated by Sascha Steinbiss almost 4 years ago

Victor Julien wrote in #note-4:

Are you able to build suri with asan enabled?

Yes, here's a run with asan:

$ sudo ./src/.libs/suricata --af-packet -c /etc/suricata/suricata-dcso.yaml --pidfile /run/suricata.pid
[31454] 28/4/2020 -- 06:54:04 - (conf-yaml-loader.c:267) <Info> (ConfYamlParse) -- Including configuration file customer-vars.yaml.
[31454] 28/4/2020 -- 06:54:04 - (conf-yaml-loader.c:267) <Info> (ConfYamlParse) -- Including configuration file dcso-rules.yaml.
[31454] 28/4/2020 -- 06:54:04 - (conf-yaml-loader.c:267) <Info> (ConfYamlParse) -- Including configuration file dcso-logging.yaml.
[31454] 28/4/2020 -- 06:54:04 - (conf-yaml-loader.c:267) <Info> (ConfYamlParse) -- Including configuration file dcso-applayer.yaml.
[31454] 28/4/2020 -- 06:54:04 - (conf-yaml-loader.c:267) <Info> (ConfYamlParse) -- Including configuration file dcso-advanced.yaml.
[31454] 28/4/2020 -- 06:54:04 - (conf-yaml-loader.c:267) <Info> (ConfYamlParse) -- Including configuration file dcso-affinity.yaml.
[31454] 28/4/2020 -- 06:54:04 - (conf-yaml-loader.c:267) <Info> (ConfYamlParse) -- Including configuration file dcso-interfaces.yaml.
[31454] 28/4/2020 -- 06:54:04 - (suricata.c:1057) <Notice> (LogVersion) -- This is Suricata version 6.0.0-dev (eef776087 2020-04-27) running in SYSTEM mode
[31454] 28/4/2020 -- 06:54:05 - (util-logopenfile.c:81) <Warning> (SCLogOpenUnixSocketFp) -- [ERRCODE: SC_ERR_SOCKET(200)] - Error connecting to socket "/tmp/files.sock": No such file or directory (will keep trying)
[31454] 28/4/2020 -- 06:59:02 - (runmode-af-packet.c:585) <Error> (ParseAFPConfig) -- [ERRCODE: SC_ERR_INVALID_VALUE(130)] - Block-size must be a multiple of pagesize.
[31454] 28/4/2020 -- 06:59:05 - (runmode-af-packet.c:585) <Error> (ParseAFPConfig) -- [ERRCODE: SC_ERR_INVALID_VALUE(130)] - Block-size must be a multiple of pagesize.
[31454] 28/4/2020 -- 06:59:09 - (tm-threads.c:1888) <Notice> (TmThreadWaitOnThreadInit) -- all 18 packet processing threads, 6 management threads initialized, engine started.
[33215] 28/4/2020 -- 06:59:12 - (util-log-redis.c:252) <Notice> (SCLogRedisWriteAsync) -- Trying to connect to Redis
[33215] 28/4/2020 -- 06:59:12 - (util-log-redis.c:134) <Notice> (SCRedisAsyncEchoCommandCallback) -- Connected to Redis.
[31454] 28/4/2020 -- 07:11:21 - (detect-engine.c:4008) <Notice> (DetectEngineReload) -- rule reload starting
AddressSanitizer:DEADLYSIGNAL
=================================================================
==31454==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000002 (pc 0x7f89bcbe2b02 bp 0x7f8b3a48e790 sp 0x7f8b3a48e710 T10)
==31454==The signal is caused by a WRITE memory access.
==31454==Hint: address points to the zero page.
AddressSanitizer:DEADLYSIGNAL
    #0 0x7f89bcbe2b01  (<unknown module>)

AddressSanitizer can not provide additional info.
AddressSanitizer:DEADLYSIGNAL
SUMMARY: AddressSanitizer: SEGV (<unknown module>) 
Thread T10 (W#01-enp175s0f1) created by T0 (Suricata-Main) here:
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer:DEADLYSIGNAL
    #0 0x7f8b4d8c5db0 in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x50db0)
    #1 0x556819944fd9 in TmThreadSpawn /home/dcsoadm/suri-upstream/src/tm-threads.c:1643
    #2 0x556819a00ef4 in RunModeSetLiveCaptureWorkersForDevice /home/dcsoadm/suri-upstream/src/util-runmodes.c:333
    #3 0x556819a010b4 in RunModeSetLiveCaptureWorkers /home/dcsoadm/suri-upstream/src/util-runmodes.c:363
    #4 0x556819895cee in RunModeIdsAFPWorkers /home/dcsoadm/suri-upstream/src/runmode-af-packet.c:909
    #5 0x5568198a1cc2 in RunModeDispatch /home/dcsoadm/suri-upstream/src/runmodes.c:377
    #6 0x556819935e70 in SuricataMain /home/dcsoadm/suri-upstream/src/suricata.c:2760
    #7 0x556819566364 in main /home/dcsoadm/suri-upstream/src/main.c:22
    #8 0x7f8b4afd709a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)

==31454==ABORTING

The bisect failure shouldn't happen anymore in the future. We added a CI check to build each commit.

Okay, thanks!

Actions

Copy link

Updated by Sascha Steinbiss almost 4 years ago

Sascha Steinbiss wrote in #note-3:

I can look deeper into how different configuration options might influence this issue later today.

Unfortunately I was unable to reproduce the problem on my laptop, even with similar config settings (I couldn't just lift over CPU affinity etc but otherwise I took the configs from the live system) and the ruleset used on the sensors.

What I did find out that might be helpful:

Deactivating the second interface in the af-packet config caused the problem to disappear.

af-packet:
  - interface: enp175s0f0
    cluster-id: 99
    cluster-type: cluster_qm
    threads: 9
    defrag: yes
    use-mmap: yes
    mmap-locked: yes
    tpacket-v3: yes
    rollover: no
    use-emergency-flush: yes
    ring-size: 200000
    block-size: 4194304
  - interface: enp175s0f1
    cluster-id: 98
    cluster-type: cluster_qm
    threads: 9
    defrag: yes
    use-mmap: yes
    mmap-locked: yes
    tpacket-v3: yes
    rollover: no
    use-emergency-flush: yes
    ring-size: 200000
    block-size: 4194304

Reverting the four commits mentioned above (and adjusting some of the code introduced after these commits) also caused the problem to not occur again, regardless of how many interfaces are active.

I'll take a closer look at these commits, but I can't see anything obvious.

Actions

Copy link

Updated by Sascha Steinbiss almost 4 years ago

Some more insight:

What I did find out that might be helpful:

Deactivating the second interface in the af-packet config caused the problem to disappear.
[...]

This is apparently not that important. The second interface was the interface carrying the traffic, the first did not see any. Hence the difference in effect. Only enabling the second interface (~9Gb/s) also leads to the problem with the original master version.

Actions

Copy link

Updated by Sascha Steinbiss almost 4 years ago

I have done some more debugging. The crash happens shortly after having called FlowWorkerReplaceDetectCtx() on the thread that later crashes.
I also noticed that undefining PCRE_HAVE_JIT and PCRE_HAVE_JIT_EXEC also fixes the problem for me -- one more indication that the issue is likely there.

Actions

Copy link