Bug #1128
closedSegmentation fault - live rule reload
Description
Segmentation fault after sending SIGUSR2 signal.
# /usr/sbin/suricata --build-info This is Suricata version 2.0rc2 RELEASE Features: NFQ PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK HAVE_NSS HAVE_LIBJANSSON SIMD support: none Atomic intrisics: 1 2 4 8 byte(s) 64-bits, Little-endian architecture GCC version 4.4.7 20120313 (Red Hat 4.4.7-4), C version 199901 compiled with -fstack-protector compiled with _FORTIFY_SOURCE=2 L1 cache line size (CLS)=64 compiled with LibHTP v0.5.10, linked against LibHTP v0.5.10 Suricata Configuration: AF_PACKET support: yes PF_RING support: no NFQueue support: yes IPFW support: no DAG enabled: no Napatech enabled: no Unix socket enabled: yes Detection enabled: yes libnss support: yes libnspr support: yes libjansson support: yes Prelude support: no PCRE jit: no libluajit: no libgeoip: yes Non-bundled htp: no Old barnyard2 support: no CUDA enabled: no Suricatasc install: yes Unit tests enabled: no Debug output enabled: no Debug validation enabled: no Profiling enabled: no Profiling locks enabled: no Coccinelle / spatch: no Generic build parameters: Installation prefix (--prefix): /usr Configuration directory (--sysconfdir): /etc/suricata/ Log directory (--localstatedir) : /var/log/suricata/ Host: x86_64-redhat-linux-gnu GCC binary: gcc -std=gnu99 GCC Protect enabled: yes GCC march native enabled: no GCC Profile enabled: no
./configure --enable-gccprotect --disable-gccmarch-native --enable-nfqueue --enable-af-packet --enable-jansson --enable-geoip CFLAGS="-ggdb -O0"
Files
Updated by Victor Julien over 10 years ago
The backtrace is very strange, it shows something that shouldn't be possible.
Thread 1 shows:
#3 0x00000000004ad503 in DetectEngineLiveRuleSwap (arg=0x0) at detect-engine.c:508
However, in no code path can 'arg' be 0x0. So either this is a really weird corruption, or the bt is generated based on the wrong core-binary combination.
Can you post the full output of gdbs bt? So everything after you started gdb, including all gdb startup output, etc.
Updated by Victor Serbu over 10 years ago
I have a problem with gdb (segfault) and i will try monday with another version of gdb (I have to leave the office right now).
I have run the following command:
usr/sbin/suricata -c /etc/suricata/suricata.yaml.minimal-rules --af-packet=eth1
- gdb /usr/sbin/suricata core.2235
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/suricata...Reading symbols from /usr/lib/debug/usr/sbin/suricata.debug...done.
done.
warning: core file may not match specified executable file.
[New Thread 2242]
[New Thread 2239]
[New Thread 2238]
[New Thread 2237]
[New Thread 2236]
[New Thread 2235]
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/03/673c781af50aad25840c1e54eb11a0165f6468
Reading symbols from /usr/lib64/libhtp-0.5.10.so.1.0.0...Reading symbols from /usr/lib/debug/usr/lib64/libhtp-0.5.10.so.1.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/libhtp-0.5.10.so.1.0.0
Reading symbols from /usr/lib64/libGeoIP.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libGeoIP.so.1
Reading symbols from /usr/lib64/libmagic.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libmagic.so.1
Reading symbols from /lib64/libcap-ng.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcap-ng.so.0
Reading symbols from /usr/lib64/libpcap.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libpcap.so.1
Reading symbols from /lib64/libnet.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnet.so.1
Reading symbols from /usr/lib64/libnetfilter_queue.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnetfilter_queue.so.1
Reading symbols from /usr/lib64/libnfnetlink.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnfnetlink.so.0
Reading symbols from /usr/lib64/libjansson.so.4...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libjansson.so.4
Reading symbols from /usr/lib64/libyaml-0.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libyaml-0.so.2
Reading symbols from /lib64/libpcre.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libpcre.so.0
Reading symbols from /usr/lib64/libssl3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl3.so
Reading symbols from /usr/lib64/libsmime3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libsmime3.so
Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /usr/lib64/libplds4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libplds4.so
Reading symbols from /usr/lib64/libplc4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libplc4.so
Reading symbols from /usr/lib64/libnspr4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnspr4.so
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: debugger service failed
warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
To enable execution of this file add
add-auto-load-safe-path /lib64/libthread_db-1.0.so
line to your configuration file "/root/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/root/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
info "(gdb)Auto-loading safe path"
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: debugger service failed
warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: debugger service failed
warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: debugger service failed
warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: debugger service failed
warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libmnl.so.0...(no debugging symbols found)...done.
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: debugger service failed
warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Loaded symbols for /usr/lib64/libmnl.so.0
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: debugger service failed
warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /usr/lib64/libsoftokn3.so...(no debugging symbols found)...done.
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: debugger service failed
warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Loaded symbols for /usr/lib64/libsoftokn3.so
Reading symbols from /usr/lib64/libsqlite3.so.0...(no debugging symbols found)...done.
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: debugger service failed
warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Loaded symbols for /usr/lib64/libsqlite3.so.0
Reading symbols from /usr/lib64/libfreebl3.so...(no debugging symbols found)...done.
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: debugger service failed
warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Loaded symbols for /usr/lib64/libfreebl3.so
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fffcd1fe000
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: debugger service failed
warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Core was generated by `/usr/sbin/suricata -c /etc/suricata/suricata.yaml.minimal-rules --af-packet=eth'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007faabf6ee183 in __strtoll_l_internal () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install GeoIP-1.4.8-1.el6.x86_64 file-libs-5.04-15.el6.x86_64 glibc-2.12-1.132.el6.x86_64 jansson-2.6-1.el6.x86_64 libcap-ng-0.6.4-3.el6_0.1.x86_64 libmnl-1.0.3-1.el6.x86_64 libnet-1.1.6-7.el6.x86_64 libnetfilter_queue-1.0.2-1.el6.x86_64 libnfnetlink-1.0.0-1.el6.x86_64 libpcap-1.4.0-1.20130826git2dbcaa1.el6.x86_64 libyaml-0.1.5-1.el6.x86_64 nspr-devel-4.10.2-1.el6_5.x86_64 nss-3.15.3-6.el6_5.x86_64 nss-softokn-3.14.3-9.el6.x86_64 nss-softokn-freebl-3.14.3-9.el6.x86_64 nss-util-3.15.3-1.el6_5.x86_64 pcre-7.8-6.el6.x86_64 sqlite-3.6.20-1.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt f
#0 0x00007faabf6ee183 in __strtoll_l_internal () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007faabf6eaf60 in atoi () from /lib64/libc.so.6
No symbol table info available.
#2 0x00000000004af1c2 in DetectEngineCtxInit () at detect-engine.c:802
de_ctx = 0x7faaa80008c0
seq_node = 0x7faaa4015eb0
insp_recursion_limit_node = 0x7faaa4015f00
de_engine_node = 0x7faaa4015680
insp_recursion_limit = 0x7faaa4015f80 "3000"
FUNCTION = "DetectEngineCtxInit"
#3 0x00000000004ad503 in DetectEngineLiveRuleSwap (arg=0x0) at detect-engine.c:508
i = 0
no_of_detect_tvs = 0
old_de_ctx = 0x0
tv = 0x0
FUNCTION = "DetectEngineLiveRuleSwap"
tv_local = 0x0
Segmentation fault
Updated by Victor Julien over 10 years ago
Sounds like this system is unreliable. I'd check the hardware and OS.
Updated by Victor Julien over 10 years ago
- Status changed from New to Assigned
- Assignee set to Victor Julien
- Target version set to 2.0rc3
I'm actually getting the same segv, including the same weird bt.
Updated by Victor Julien over 10 years ago
- Status changed from Assigned to Closed
- % Done changed from 0 to 100
Updated by jason jones over 10 years ago
Victor Julien wrote:
I'm testing your git repo with this fix and am still getting a segfault. should a new issue be opened or this one re-opened?
The backtrace i get in gdb is:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff34e5700 (LWP 37953)]
0x000000000047a527 in DetectEngineLiveRuleSwap (arg=<optimized out>) at detect-engine.c:678
678 old_de_ctx = old_det_ctx0->de_ctx;
(gdb) bt full
#0 0x000000000047a527 in DetectEngineLiveRuleSwap (arg=<optimized out>) at detect-engine.c:678
i = <optimized out>
no_of_detect_tvs = <optimized out>
old_de_ctx = 0x0
tv = <optimized out>
FUNCTION = "DetectEngineLiveRuleSwap"
tv_local = <optimized out>
old_det_ctx = 0x7ffff34e4500
new_det_ctx = 0x4d414a0
detect_tvs = 0x7ffff34e44c0
de_ctx = 0x4d414a0