Project

General

Profile

Actions
Copy link

Bug #1669

closed

Suricate 3.0RC3 segfault after 10 hours

Added by Anonymous over 8 years ago. Updated about 8 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Victor Julien
Target version:
3.0.1RC1
Affected Versions:
Effort:
Difficulty:
Label:

Description

Daily a service restart at 07h CET (logrotate and rules), and sometimes at 16h Suricata segfaults.

Kernel ring message:

RxPFReth51[38079]: segfault at 7f43a1975000 ip 00000000005930c9 sp 00007f43a2373420 error 4 in suricata[400000+225000]

Redhat 6.7

2.6.32-573.12.1.el6.x86_64 #1 SMP Mon Nov 23 12:55:32 EST 2015 x86_64 x86_64 x86_64 GNU/Linux

Pfring from source:
~]# cat /proc/net/pf_ring/info

PF_RING Version          : 6.3.0 (unknown)
Total rings              : 4

Standard (non DNA/ZC) Options
Ring slots               : 8192
Slot version             : 16
Capture TX               : No [RX only]
IP Defragment            : No
Socket Mode              : Standard
Total plugins            : 0
Cluster Fragment Queue   : 0
Cluster Fragment Discard : 0

Suricata:

~]# ldd /usr/bin/suricata
    linux-vdso.so.1 =>  (0x00007ffc3298f000)
    libhtp-0.5.18.so.1 => /usr/lib/libhtp-0.5.18.so.1 (0x0000003625800000)
    libGeoIP.so.1 => /usr/lib64/libGeoIP.so.1 (0x0000003ee0200000)
    libluajit-5.1.so.2 => /usr/local/lib/libluajit-5.1.so.2 (0x00007ff5315eb000)
    libmagic.so.1 => /usr/local/lib/libmagic.so.1 (0x00007ff5313ce000)
    libcap-ng.so.0 => /lib64/libcap-ng.so.0 (0x0000003219800000)
    libpfring.so => /usr/local/lib/libpfring.so (0x00007ff53116f000)
    libpcap.so.1 => /usr/local/lib/libpcap.so.1 (0x00007ff530eda000)
    libnet.so.1 => /lib64/libnet.so.1 (0x0000003219c00000)
    libjansson.so.4 => /usr/lib64/libjansson.so.4 (0x000000321ac00000)
    libyaml-0.so.2 => /usr/lib64/libyaml-0.so.2 (0x000000321a400000)
    libpcre.so.1 => /opt/pcre-8.37/lib/libpcre.so.1 (0x00007ff530c6e000)
    librt.so.1 => /lib64/librt.so.1 (0x0000003218800000)
    libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x0000003219400000)
    libssl3.so => /usr/lib64/libssl3.so (0x0000003ce4e00000)
    libsmime3.so => /usr/lib64/libsmime3.so (0x0000003ce5200000)
    libnss3.so => /usr/lib64/libnss3.so (0x0000003ce4a00000)
    libnssutil3.so => /usr/lib64/libnssutil3.so (0x0000003815e00000)
    libplds4.so => /lib64/libplds4.so (0x0000003816600000)
    libplc4.so => /lib64/libplc4.so (0x0000003816200000)
    libnspr4.so => /lib64/libnspr4.so (0x0000003815a00000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003218400000)
    libdl.so.2 => /lib64/libdl.so.2 (0x0000003218000000)
    libc.so.6 => /lib64/libc.so.6 (0x0000003217c00000)
    libz.so.1 => /lib64/libz.so.1 (0x0000003219000000)
    libm.so.6 => /lib64/libm.so.6 (0x0000003218c00000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x000000321a000000)
    /lib64/ld-linux-x86-64.so.2 (0x0000003217800000)

~]# suricata --build-info

This is Suricata version 3.0RC3 RELEASE
Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 PF_RING AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LUA HAVE_LUAJIT HAVE_LIBJANSSON TLS
SIMD support: SSE_4_2 SSE_4_1 SSE_3
Atomic intrisics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 4.4.7 20120313 (Red Hat 4.4.7-16), C version 199901
compiled with -fstack-protector
compiled with _FORTIFY_SOURCE=2
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.18, linked against LibHTP v0.5.18

Suricata Configuration:
  AF_PACKET support:                       yes
  PF_RING support:                         yes
  NFQueue support:                         no
  NFLOG support:                           no
  IPFW support:                            no
  Netmap support:                          no
  DAG enabled:                             no
  Napatech enabled:                        no

  Unix socket enabled:                     yes
  Detection enabled:                       yes

  libnss support:                          yes
  libnspr support:                         yes
  libjansson support:                      yes
  hiredis support:                         no
  Prelude support:                         no
  PCRE jit:                                yes
  LUA support:                             yes, through luajit
  libluajit:                               yes
  libgeoip:                                yes
  Non-bundled htp:                         no
  Old barnyard2 support:                   yes
  CUDA enabled:                            no

  Suricatasc install:                      yes

  Unit tests enabled:                      no
  Debug output enabled:                    no
  Debug validation enabled:                no
  Profiling enabled:                       no
  Profiling locks enabled:                 no
  Coccinelle / spatch:                     no

Generic build parameters:
  Installation prefix:                     /usr
  Configuration directory:                 /etc/suricata/
  Log directory:                           /var/log/suricata/

  --prefix                                 /usr
  --sysconfdir                             /etc
  --localstatedir                          /var

  Host:                                    x86_64-unknown-linux-gnu
  Compiler:                                gcc (exec name) / gcc (real)
  GCC Protect enabled:                     yes
  GCC march native enabled:                yes
  GCC Profile enabled:                     no
  Position Independent Executable enabled: no
  CFLAGS                                   -g -O2 -march=native
  PCAP_CFLAGS                               -I/usr/local/include
  SECCFLAGS                                -fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security

Actions
Copy link
 #1

Updated by Peter Manev over 8 years ago

From the build-info output I see that there is no debug enabled - hence most likely not so much useful information to chase the issue.
Since you mention this is reproducible - can you please recompile with - 

     CFLAGS="-O0 -ggdb"  ./configure.........

Then the coredump would be very helpful.

Actions
Copy link
 #2

Updated by Anonymous over 8 years ago

Peter Manev wrote:

From the build-info output I see that there is no debug enabled - hence most likely not so much useful information to chase the issue.
Since you mention this is reproducible - can you please recompile with -
[...]

Then the coredump would be very helpful.

Thanks for the input, just reconfigured and compiled/linked again and restart suricata. See what happens coming days.
Host: x86_64-unknown-linux-gnu
Compiler: gcc (exec name) / gcc (real)
GCC Protect enabled: yes
GCC march native enabled: yes
GCC Profile enabled: no
Position Independent Executable enabled: no
CFLAGS -O0 -ggdb -march=native
PCAP_CFLAGS -I/usr/local/include
SECCFLAGS -fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security

Actions
Copy link
 #3

Updated by Andreas Moe about 8 years ago

Any news for this issue? Haven't seen the same issue but very anxious for the outcome / resolution.

Actions
Copy link
 #5

Updated by Victor Julien about 8 years ago

  • Target version changed from 3.0 to 70
Actions
Copy link
 #6

Updated by Anonymous about 8 years ago

Well since recompiling it as requested with CFLAGS="-O0 ggdb" I've not seen any crash yet 8( Did not upgrade poring or kernel in-between so a little mytery.

Actions
Copy link
 #7

Updated by Victor Julien about 8 years ago

Andre, do you happen to have some of the nic offloading features still on? Can you show output of 'ethtool -k <youriface>'?

Actions
Copy link
 #8

Updated by Anonymous about 8 years ago

Hello Victor,

Suricata uses input from 2 nic's (--pfring-int=eth4 --pfring-int=eth5)

  1. ethtool -k eth4
    Features for eth4:
    rx-checksumming: on
    tx-checksumming: on
    tx-checksum-ipv4: on
    tx-checksum-unneeded: off [fixed]
    tx-checksum-ip-generic: off [fixed]
    tx-checksum-ipv6: on
    tx-checksum-fcoe-crc: on [fixed]
    tx-checksum-sctp: on
    scatter-gather: on
    tx-scatter-gather: on
    tx-scatter-gather-fraglist: off [fixed]
    tcp-segmentation-offload: on
    tx-tcp-segmentation: on
    tx-tcp-ecn-segmentation: off [fixed]
    tx-tcp6-segmentation: on
    udp-fragmentation-offload: off [fixed]
    generic-segmentation-offload: on
    generic-receive-offload: on
    large-receive-offload: on
    rx-vlan-offload: on
    tx-vlan-offload: on
    ntuple-filters: off
    receive-hashing: on
    highdma: on [fixed]
    rx-vlan-filter: on [fixed]
    vlan-challenged: off [fixed]
    tx-lockless: off [fixed]
    netns-local: off [fixed]
    tx-gso-robust: off [fixed]
    tx-fcoe-segmentation: on [fixed]
    tx-gre-segmentation: off [fixed]
    tx-udp_tnl-segmentation: off [fixed]
    fcoe-mtu: off [fixed]
    loopback: off [fixed]
  1. ethtool -k eth5
    Features for eth5:
    rx-checksumming: on
    tx-checksumming: on
    tx-checksum-ipv4: on
    tx-checksum-unneeded: off [fixed]
    tx-checksum-ip-generic: off [fixed]
    tx-checksum-ipv6: on
    tx-checksum-fcoe-crc: on [fixed]
    tx-checksum-sctp: on
    scatter-gather: on
    tx-scatter-gather: on
    tx-scatter-gather-fraglist: off [fixed]
    tcp-segmentation-offload: on
    tx-tcp-segmentation: on
    tx-tcp-ecn-segmentation: off [fixed]
    tx-tcp6-segmentation: on
    udp-fragmentation-offload: off [fixed]
    generic-segmentation-offload: on
    generic-receive-offload: on
    large-receive-offload: on
    rx-vlan-offload: on
    tx-vlan-offload: on
    ntuple-filters: off
    receive-hashing: on
    highdma: on [fixed]
    rx-vlan-filter: on [fixed]
    vlan-challenged: off [fixed]
    tx-lockless: off [fixed]
    netns-local: off [fixed]
    tx-gso-robust: off [fixed]
    tx-fcoe-segmentation: on [fixed]
    tx-gre-segmentation: off [fixed]
    tx-udp_tnl-segmentation: off [fixed]
    fcoe-mtu: off [fixed]
    loopback: off [fixed]
Actions
Copy link
 #9

Updated by Anonymous about 8 years ago

Crash today at 13:09 CET, running since a service restart at 07:00 CET today.

RxPFReth5160377: segfault at 7fe120109000 ip 000000000061980b sp 00007fe120b072f0 error 4 in suricata[400000+2cb000]

]# ethtool -k eth5
Features for eth5:
rx-checksumming: off
tx-checksumming: off
tx-checksum-ipv4: off
tx-checksum-unneeded: off [fixed]
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: off
tx-checksum-fcoe-crc: on [fixed]
tx-checksum-sctp: off
scatter-gather: off
tx-scatter-gather: off
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp6-segmentation: off
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: off
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: on [fixed]
tx-gre-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
fcoe-mtu: off [fixed]
loopback: off [fixed]

]# suricata --build-info
This is Suricata version 3.0 RELEASE
Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 PF_RING AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LUA HAVE_LUAJIT HAVE_LIBJANSSON TLS
SIMD support: SSE_4_2 SSE_4_1 SSE_3
Atomic intrisics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 4.4.7 20120313 (Red Hat 4.4.7-16), C version 199901
compiled with -fstack-protector
compiled with _FORTIFY_SOURCE=2
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.18, linked against LibHTP v0.5.18

Suricata Configuration:
AF_PACKET support: yes
PF_RING support: yes
NFQueue support: no
NFLOG support: no
IPFW support: no
Netmap support: no
DAG enabled: no
Napatech enabled: no

Unix socket enabled:                     yes
  Detection enabled:                       yes
libnss support:                          yes
  libnspr support:                         yes
  libjansson support:                      yes
  hiredis support:                         no
  Prelude support:                         no
  PCRE jit:                                yes
  LUA support:                             yes, through luajit
  libluajit:                               yes
  libgeoip:                                yes
  Non-bundled htp:                         no
  Old barnyard2 support:                   yes
  CUDA enabled:                            no
Suricatasc install:                      yes
Unit tests enabled:                      no
  Debug output enabled:                    no
  Debug validation enabled:                no
  Profiling enabled:                       no
  Profiling locks enabled:                 no
  Coccinelle / spatch:                     no

Generic build parameters:
Installation prefix: /usr
Configuration directory: /etc/suricata/
Log directory: /var/log/suricata/

--prefix                                 /usr
  --sysconfdir                             /etc
  --localstatedir                          /var
Host:                                    x86_64-unknown-linux-gnu
  Compiler:                                gcc (exec name) / gcc (real)
  GCC Protect enabled:                     yes
  GCC march native enabled:                yes
  GCC Profile enabled:                     no
  Position Independent Executable enabled: no
  CFLAGS                                   -O0 -ggdb -march=native
  PCAP_CFLAGS                               -I/usr/local/include
  SECCFLAGS                                -fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security

According to CFLAGS a crashdump should be generated , correct? But where to find it?

Actions
Copy link
 #10

Updated by Victor Julien about 8 years ago

  • Description updated (diff)
Actions
Copy link
 #11

Updated by Victor Julien about 8 years ago

  • Status changed from New to Assigned
  • Assignee set to Victor Julien

Still investigating.

Actions
Copy link
 #12

Updated by Victor Julien about 8 years ago

  • Status changed from Assigned to Closed
  • Target version changed from 70 to 3.0.1RC1
Actions
Copy link

Also available in: Atom PDF