Actions
Bug #2953
closedbypass keyword: Suricata 4.1.x Segmentation Faults
Affected Versions:
Effort:
Difficulty:
Label:
Needs backport
Description
Been seeing this consistently on some of our higher traffic sensors since our major upgrade from Suricata 3.0.2 to 4.1.2 release and I also confirmed the issue persists on today's 4.1.4 release. Output below is from our compiled debug version. Happy to provide the crash file / core dump to a developer if needed.
Platform: Ubuntu 16.04 LTS
Uname: Linux sensor 4.4.0-142-generic #168-Ubuntu SMP Wed Jan 16 21:00:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Configure:
./configure --prefix=/usr/ --sysconfdir=/etc/ --localstatedir=/var/ --enable-geoip --enable-luajit --enable-pie --enable-hiredis --enable-gccprotect --disable-gccmarch-native --enable-debug CFLAGS="-ggdb -O0" && make && make install
Build Info and Crash Logs
user@sensor:~$ suricata --build-info
This is Suricata version 4.1.4 RELEASE
Features: DEBUG PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LUA HAVE_LUAJIT HAVE_LIBJANSSON TLS MAGIC RUST
SIMD support: none
Atomic intrisics: 1 2 4 8 byte(s)
64-bits, Little-endian architecture
GCC version 5.4.0 20160609, C version 199901
compiled with -fstack-protector
compiled with _FORTIFY_SOURCE=2
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.30, linked against LibHTP v0.5.30
Suricata Configuration:
AF_PACKET support: yes
eBPF support: no
XDP support: no
PF_RING support: no
NFQueue support: no
NFLOG support: no
IPFW support: no
Netmap support: no
DAG enabled: no
Napatech enabled: no
WinDivert enabled: no
Unix socket enabled: yes
Detection enabled: yes
Libmagic support: yes
libnss support: yes
libnspr support: yes
libjansson support: yes
liblzma support: yes
hiredis support: yes
hiredis async with libevent: yes
Prelude support: no
PCRE jit: yes
LUA support: yes, through luajit
libluajit: yes
libgeoip: yes
Non-bundled htp: no
Old barnyard2 support: no
Hyperscan support: yes
Libnet support: yes
liblz4 support: yes
Rust support: yes (default)
Rust strict mode: no
Rust debug mode: no
Rust compiler: rustc 1.31.0
Rust cargo: cargo 1.31.0
Install suricatasc: yes
Install suricata-update: yes
Profiling enabled: no
Profiling locks enabled: no
Development settings:
Coccinelle / spatch: no
Unit tests enabled: no
Debug output enabled: yes
Debug validation enabled: no
Generic build parameters:
Installation prefix: /usr
Configuration directory: /etc/suricata/
Log directory: /var/log/suricata/
--prefix /usr
--sysconfdir /etc
--localstatedir /var
--datarootdir /usr/share
Host: x86_64-pc-linux-gnu
Compiler: gcc (exec name) / gcc (real)
GCC Protect enabled: yes
GCC march native enabled: no
GCC Profile enabled: no
Position Independent Executable enabled: yes
CFLAGS -ggdb -O0 -I${srcdir}/../rust/gen/c-headers
PCAP_CFLAGS -I/usr/include
SECCFLAGS -fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security
user@sensor:~$ sudo journalctl -u suricata -ef | grep -A20 -B20 -i segv
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - Entering ... >>
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - Returning ... <<
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - Returning ... <<
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - Entering ... >>
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - Returning ... <<
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - Entering ... >>
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - Returning ... <<
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - Returning ... <<
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - type "host" name "tag" size "8"
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - type "host" name "threshold" size "8"
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - type "host" name "bit" size "8"
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - type "flow" name "expectation" size "8"
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - type "flow" name "tag" size "8"
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - type "ippair" name "expectation" size "8"
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - type "ippair" name "threshold" size "8"
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - type "ippair" name "bit" size "8"
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - Parent is waiting for child to be ready
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - Daemon: Parent waiting for child to be ready...
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - failed to lookup configuration parameter 'daemon-directory'
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - Child is ready, parent exiting
Apr 30 19:02:53 sensor systemd[1]: suricata.service: Main process exited, code=killed, status=11/SEGV
Apr 30 19:02:53 sensor systemd[1]: suricata.service: Control process exited, code=exited status=1
Apr 30 19:02:53 sensor kill[27690]: Usage:
Apr 30 19:02:53 sensor kill[27690]: kill [options] <pid> [...]
Apr 30 19:02:53 sensor kill[27690]: Options:
Apr 30 19:02:53 sensor kill[27690]: <pid> [...] send signal to every <pid> listed
Apr 30 19:02:53 sensor kill[27690]: -<signal>, -s, --signal <signal>
Apr 30 19:02:53 sensor kill[27690]: specify the <signal> to be sent
Apr 30 19:02:53 sensor kill[27690]: -l, --list=[<signal>] list all signal names, or convert one to a name
Apr 30 19:02:53 sensor kill[27690]: -L, --table list all signal names in a nice table
Apr 30 19:02:53 sensor kill[27690]: -h, --help display this help and exit
Apr 30 19:02:53 sensor kill[27690]: -V, --version output version information and exit
Apr 30 19:02:53 sensor kill[27690]: For more details see kill(1).
Apr 30 19:02:53 sensor systemd[1]: suricata.service: Unit entered failed state.
Apr 30 19:02:53 sensor systemd[1]: suricata.service: Triggering OnFailure= dependencies.
Apr 30 19:02:53 sensor systemd[1]: suricata.service: Failed with result 'exit-code'.
Apr 30 19:02:53 sensor systemd[1]: suricata.service: Service hold-off time over, scheduling restart.
Apr 30 19:02:53 sensor systemd[1]: Stopped Suricata Daemon.
Apr 30 19:02:53 sensor systemd[1]: Starting Suricata Daemon...
Apr 30 19:02:53 sensor genconf_suricata[27699]: Generating Variable File
Apr 30 19:02:53 sensor genconf_suricata[27699]: Writing to /etc/suricata/suricata_vars.yaml
user@sensor:~$ dmesg | grep seg
[70850.864982] W#01-br0[27147]: segfault at 40 ip 000055a99aa623f7 sp 00007fd16e63d5d8 error 4 in suricata-4.1.4-prod[55a99a9c6000+4b7000]
[70989.001085] W#02-br0[27582]: segfault at 40 ip 000055d9aca963f7 sp 00007f71e67545d8 error 4 in suricata-4.1.4-prod[55d9ac9fa000+4b7000]
[71083.258640] W#02-br0[27672]: segfault at 40 ip 000055ce734ba68c sp 00007f3d26afe560 error 4 in suricata-4.1.4-dbg[55ce73368000+bae000]
[71178.483007] W#01-br0[27715]: segfault at 40 ip 000056092ef8168c sp 00007fb9e4ee4560 error 4 in suricata-4.1.4-dbg[56092ee2f000+bae000]
[71224.431899] W#01-br0[27997]: segfault at 40 ip 000055ab74e8968c sp 00007f3b3854d560 error 4 in suricata-4.1.4-dbg[55ab74d37000+bae000]
[71427.648879] W#01-br0[28119]: segfault at 40 ip 000055c70935c68c sp 00007fea065ee560 error 4 in suricata-4.1.4-dbg[55c70920a000+bae000]
user@sensor:~$ tail /var/log/apport.log -n1
ERROR: apport (pid 28453) Tue Apr 30 19:09:42 2019: apport: report /var/crash/_usr_bin_suricata-4.1.4-dbg.0.crash already exists and unseen, doing nothing to avoid disk usage DoS
Actions