Project

General

Profile

Actions

Bug #1128

closed

Segmentation fault - live rule reload

Added by Victor Serbu over 10 years ago. Updated over 10 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

Segmentation fault after sending SIGUSR2 signal.

# /usr/sbin/suricata --build-info
This is Suricata version 2.0rc2 RELEASE
Features: NFQ PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK HAVE_NSS HAVE_LIBJANSSON
SIMD support: none
Atomic intrisics: 1 2 4 8 byte(s)
64-bits, Little-endian architecture
GCC version 4.4.7 20120313 (Red Hat 4.4.7-4), C version 199901
compiled with -fstack-protector
compiled with _FORTIFY_SOURCE=2
L1 cache line size (CLS)=64
compiled with LibHTP v0.5.10, linked against LibHTP v0.5.10
Suricata Configuration:
  AF_PACKET support:                       yes
  PF_RING support:                         no
  NFQueue support:                         yes
  IPFW support:                            no
  DAG enabled:                             no
  Napatech enabled:                        no
  Unix socket enabled:                     yes
  Detection enabled:                       yes

  libnss support:                          yes
  libnspr support:                         yes
  libjansson support:                      yes
  Prelude support:                         no
  PCRE jit:                                no
  libluajit:                               no
  libgeoip:                                yes
  Non-bundled htp:                         no
  Old barnyard2 support:                   no
  CUDA enabled:                            no

  Suricatasc install:                      yes

  Unit tests enabled:                      no
  Debug output enabled:                    no
  Debug validation enabled:                no
  Profiling enabled:                       no
  Profiling locks enabled:                 no
  Coccinelle / spatch:                     no

Generic build parameters:
  Installation prefix (--prefix):          /usr
  Configuration directory (--sysconfdir):  /etc/suricata/
  Log directory (--localstatedir) :        /var/log/suricata/

  Host:                                    x86_64-redhat-linux-gnu
  GCC binary:                              gcc -std=gnu99
  GCC Protect enabled:                     yes
  GCC march native enabled:                no
  GCC Profile enabled:                     no
./configure --enable-gccprotect --disable-gccmarch-native --enable-nfqueue --enable-af-packet  --enable-jansson --enable-geoip CFLAGS="-ggdb -O0" 

Files

gdb-liverulereload.txt (1.95 KB) gdb-liverulereload.txt Victor Serbu, 03/07/2014 07:58 AM
Actions #1

Updated by Victor Julien over 10 years ago

  • Description updated (diff)
Actions #2

Updated by Victor Julien over 10 years ago

The backtrace is very strange, it shows something that shouldn't be possible.

Thread 1 shows:

#3  0x00000000004ad503 in DetectEngineLiveRuleSwap (arg=0x0) at detect-engine.c:508

However, in no code path can 'arg' be 0x0. So either this is a really weird corruption, or the bt is generated based on the wrong core-binary combination.

Can you post the full output of gdbs bt? So everything after you started gdb, including all gdb startup output, etc.

Actions #3

Updated by Victor Serbu over 10 years ago

I have a problem with gdb (segfault) and i will try monday with another version of gdb (I have to leave the office right now).

I have run the following command:
usr/sbin/suricata -c /etc/suricata/suricata.yaml.minimal-rules --af-packet=eth1

This is the partial output of gdb:
  1. gdb /usr/sbin/suricata core.2235
    GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
    Copyright (C) 2010 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law. Type "show copying"
    and "show warranty" for details.
    This GDB was configured as "x86_64-redhat-linux-gnu".
    For bug reporting instructions, please see:
    <http://www.gnu.org/software/gdb/bugs/&gt;...
    Reading symbols from /usr/sbin/suricata...Reading symbols from /usr/lib/debug/usr/sbin/suricata.debug...done.
    done.

warning: core file may not match specified executable file.
[New Thread 2242]
[New Thread 2239]
[New Thread 2238]
[New Thread 2237]
[New Thread 2236]
[New Thread 2235]
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/03/673c781af50aad25840c1e54eb11a0165f6468
Reading symbols from /usr/lib64/libhtp-0.5.10.so.1.0.0...Reading symbols from /usr/lib/debug/usr/lib64/libhtp-0.5.10.so.1.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/libhtp-0.5.10.so.1.0.0
Reading symbols from /usr/lib64/libGeoIP.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libGeoIP.so.1
Reading symbols from /usr/lib64/libmagic.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libmagic.so.1
Reading symbols from /lib64/libcap-ng.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcap-ng.so.0
Reading symbols from /usr/lib64/libpcap.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libpcap.so.1
Reading symbols from /lib64/libnet.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnet.so.1
Reading symbols from /usr/lib64/libnetfilter_queue.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnetfilter_queue.so.1
Reading symbols from /usr/lib64/libnfnetlink.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnfnetlink.so.0
Reading symbols from /usr/lib64/libjansson.so.4...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libjansson.so.4
Reading symbols from /usr/lib64/libyaml-0.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libyaml-0.so.2
Reading symbols from /lib64/libpcre.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libpcre.so.0
Reading symbols from /usr/lib64/libssl3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl3.so
Reading symbols from /usr/lib64/libsmime3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libsmime3.so
Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /usr/lib64/libplds4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libplds4.so
Reading symbols from /usr/lib64/libplc4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libplc4.so
Reading symbols from /usr/lib64/libnspr4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnspr4.so
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: debugger service failed

warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
To enable execution of this file add
add-auto-load-safe-path /lib64/libthread_db-1.0.so
line to your configuration file "/root/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/root/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
info "(gdb)Auto-loading safe path"

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: debugger service failed

warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: debugger service failed

warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: debugger service failed

warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: debugger service failed

warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libmnl.so.0...(no debugging symbols found)...done.
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: debugger service failed

warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Loaded symbols for /usr/lib64/libmnl.so.0
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: debugger service failed

warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /usr/lib64/libsoftokn3.so...(no debugging symbols found)...done.
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: debugger service failed

warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Loaded symbols for /usr/lib64/libsoftokn3.so
Reading symbols from /usr/lib64/libsqlite3.so.0...(no debugging symbols found)...done.
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: debugger service failed

warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Loaded symbols for /usr/lib64/libsqlite3.so.0
Reading symbols from /usr/lib64/libfreebl3.so...(no debugging symbols found)...done.
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: debugger service failed

warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Loaded symbols for /usr/lib64/libfreebl3.so

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fffcd1fe000
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: debugger service failed

warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Core was generated by `/usr/sbin/suricata -c /etc/suricata/suricata.yaml.minimal-rules --af-packet=eth'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007faabf6ee183 in __strtoll_l_internal () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install GeoIP-1.4.8-1.el6.x86_64 file-libs-5.04-15.el6.x86_64 glibc-2.12-1.132.el6.x86_64 jansson-2.6-1.el6.x86_64 libcap-ng-0.6.4-3.el6_0.1.x86_64 libmnl-1.0.3-1.el6.x86_64 libnet-1.1.6-7.el6.x86_64 libnetfilter_queue-1.0.2-1.el6.x86_64 libnfnetlink-1.0.0-1.el6.x86_64 libpcap-1.4.0-1.20130826git2dbcaa1.el6.x86_64 libyaml-0.1.5-1.el6.x86_64 nspr-devel-4.10.2-1.el6_5.x86_64 nss-3.15.3-6.el6_5.x86_64 nss-softokn-3.14.3-9.el6.x86_64 nss-softokn-freebl-3.14.3-9.el6.x86_64 nss-util-3.15.3-1.el6_5.x86_64 pcre-7.8-6.el6.x86_64 sqlite-3.6.20-1.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt f
#0 0x00007faabf6ee183 in _
_strtoll_l_internal () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007faabf6eaf60 in atoi () from /lib64/libc.so.6
No symbol table info available.
#2 0x00000000004af1c2 in DetectEngineCtxInit () at detect-engine.c:802
de_ctx = 0x7faaa80008c0
seq_node = 0x7faaa4015eb0
insp_recursion_limit_node = 0x7faaa4015f00
de_engine_node = 0x7faaa4015680
insp_recursion_limit = 0x7faaa4015f80 "3000"
FUNCTION = "DetectEngineCtxInit"
#3 0x00000000004ad503 in DetectEngineLiveRuleSwap (arg=0x0) at detect-engine.c:508
i = 0
no_of_detect_tvs = 0
old_de_ctx = 0x0
tv = 0x0
FUNCTION = "DetectEngineLiveRuleSwap"
tv_local = 0x0
Segmentation fault

Actions #4

Updated by Victor Julien over 10 years ago

Sounds like this system is unreliable. I'd check the hardware and OS.

Actions #5

Updated by Victor Julien over 10 years ago

  • Status changed from New to Assigned
  • Assignee set to Victor Julien
  • Target version set to 2.0rc3

I'm actually getting the same segv, including the same weird bt.

Actions #6

Updated by Victor Julien over 10 years ago

  • Status changed from Assigned to Closed
  • % Done changed from 0 to 100
Actions #7

Updated by jason jones over 10 years ago

Victor Julien wrote:

Fixed by https://github.com/inliniac/suricata/pull/883

I'm testing your git repo with this fix and am still getting a segfault. should a new issue be opened or this one re-opened?

The backtrace i get in gdb is:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff34e5700 (LWP 37953)]
0x000000000047a527 in DetectEngineLiveRuleSwap (arg=<optimized out>) at detect-engine.c:678
678 old_de_ctx = old_det_ctx0->de_ctx;
(gdb) bt full
#0 0x000000000047a527 in DetectEngineLiveRuleSwap (arg=<optimized out>) at detect-engine.c:678
i = <optimized out>
no_of_detect_tvs = <optimized out>
old_de_ctx = 0x0
tv = <optimized out>
FUNCTION = "DetectEngineLiveRuleSwap"
tv_local = <optimized out>
old_det_ctx = 0x7ffff34e4500
new_det_ctx = 0x4d414a0
detect_tvs = 0x7ffff34e44c0
de_ctx = 0x4d414a0

Actions #8

Updated by Victor Julien over 10 years ago

Lets create a new ticket.

Actions

Also available in: Atom PDF