Project

General

Profile

Actions

Bug #3488

closed

Memory leak in 5.0.2

Added by Vlad Grigorescu about 4 years ago. Updated about 2 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
Affected Versions:
Effort:
Difficulty:
Label:

Description

We were seeing a bad memory leak in Suricata 5.0.0, and upgraded to 5.0.2. The situation is significantly better, but there's still a memory leak as compared with 4.1.3.

I'm attaching screenshots of memory usage over 1 week from two systems looking at very similar traffic. For the system running Suricata v5, you can see when we upgraded from 5.0.0 to 5.0.2 on 2/13. However, the memory use is slowly creeping up.

suricata --build-info from both systems follow:

This is Suricata version 5.0.2 RELEASE
Features: PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LUA HAVE_LUAJIT HAVE_LIBJANSSON TLS MAGIC RUST
SIMD support: SSE_3
Atomic intrinsics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 4.8.5 20150623 (Red Hat 4.8.5-39), C version 199901
compiled with _FORTIFY_SOURCE=0
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.32, linked against LibHTP v0.5.32

Suricata Configuration:
  AF_PACKET support:                       yes
  eBPF support:                            no
  XDP support:                             no
  PF_RING support:                         no
  NFQueue support:                         no
  NFLOG support:                           no
  IPFW support:                            no
  Netmap support:                          no
  DAG enabled:                             no
  Napatech enabled:                        no
  WinDivert enabled:                       no

  Unix socket enabled:                     yes
  Detection enabled:                       yes

  Libmagic support:                        yes
  libnss support:                          yes
  libnspr support:                         yes
  libjansson support:                      yes
  hiredis support:                         yes
  hiredis async with libevent:             no
  Prelude support:                         no
  PCRE jit:                                yes
  LUA support:                             yes, through luajit
  libluajit:                               yes
  GeoIP2 support:                          yes
  Non-bundled htp:                         no
  Old barnyard2 support:                   no
  Hyperscan support:                       no
  Libnet support:                          yes
  liblz4 support:                          yes

  Rust support:                            yes
  Rust strict mode:                        no
  Rust compiler path:                      /usr/bin/rustc
  Rust compiler version:                   rustc 1.38.0
  Cargo path:                              /usr/bin/cargo
  Cargo version:                           cargo 1.38.0
  Cargo vendor:                            yes

  Python support:                          yes
  Python path:                             /usr/bin/python2.7
  Python distutils                         yes
  Python yaml                              yes
  Install suricatactl:                     yes
  Install suricatasc:                      yes
  Install suricata-update:                 yes

  Profiling enabled:                       no
  Profiling locks enabled:                 no

Development settings:
  Coccinelle / spatch:                     no
  Unit tests enabled:                      no
  Debug output enabled:                    no
  Debug validation enabled:                no

Generic build parameters:
  Installation prefix:                     /usr/local
  Configuration directory:                 /usr/local/etc/suricata/
  Log directory:                           /usr/local/var/log/suricata/

  --prefix                                 /usr/local
  --sysconfdir                             /usr/local/etc
  --localstatedir                          /usr/local/var
  --datarootdir                            /usr/local/share

  Host:                                    x86_64-pc-linux-gnu
  Compiler:                                gcc (exec name) / gcc (real)
  GCC Protect enabled:                     no
  GCC march native enabled:                yes
  GCC Profile enabled:                     no
  Position Independent Executable enabled: no
  CFLAGS                                   -g -O2 -march=native -I${srcdir}/../rust/gen/c-headers
  PCAP_CFLAGS
  SECCFLAGS

--------------------------------------
This is Suricata version 4.1.3 RELEASE
Features: PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LUA HAVE_LUAJIT HAVE_LIBJANSSON TLS MAGIC RUST
SIMD support: SSE_4_2 SSE_4_1 SSE_3
Atomic intrisics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 4.8.5 20150623 (Red Hat 4.8.5-36), C version 199901
compiled with _FORTIFY_SOURCE=0
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.30, linked against LibHTP v0.5.30

Suricata Configuration:
  AF_PACKET support:                       yes
  eBPF support:                            no
  XDP support:                             no
  PF_RING support:                         no
  NFQueue support:                         no
  NFLOG support:                           no
  IPFW support:                            no
  Netmap support:                          no
  DAG enabled:                             no
  Napatech enabled:                        no
  WinDivert enabled:                       no

  Unix socket enabled:                     yes
  Detection enabled:                       yes

  Libmagic support:                        yes
  libnss support:                          yes
  libnspr support:                         yes
  libjansson support:                      yes
  liblzma support:                         yes
  hiredis support:                         yes
  hiredis async with libevent:             no
  Prelude support:                         no
  PCRE jit:                                yes
  LUA support:                             yes, through luajit
  libluajit:                               yes
  libgeoip:                                yes
  Non-bundled htp:                         no
  Old barnyard2 support:                   no
  Hyperscan support:                       no
  Libnet support:                          yes
  liblz4 support:                          yes

  Rust support:                            yes (default)
  Rust strict mode:                        no
  Rust debug mode:                         no
  Rust compiler:                           rustc 1.36.0
  Rust cargo:                              cargo 1.36.0

  Install suricatasc:                      yes
  Install suricata-update:                 yes

  Profiling enabled:                       no
  Profiling locks enabled:                 no

Development settings:
  Coccinelle / spatch:                     no
  Unit tests enabled:                      no
  Debug output enabled:                    no
  Debug validation enabled:                no

Generic build parameters:
  Installation prefix:                     /usr/local/security/suricata/builds/suricata_4.1.3
  Configuration directory:                 /usr/local/security/suricata/builds/suricata_4.1.3/etc/suricata/
  Log directory:                           /usr/local/security/suricata/builds/suricata_4.1.3/var/log/suricata/

  --prefix                                 /usr/local/security/suricata/builds/suricata_4.1.3
  --sysconfdir                             /usr/local/security/suricata/builds/suricata_4.1.3/etc
  --localstatedir                          /usr/local/security/suricata/builds/suricata_4.1.3/var
  --datarootdir                            /usr/local/security/suricata/builds/suricata_4.1.3/share

  Host:                                    x86_64-pc-linux-gnu
  Compiler:                                gcc (exec name) / gcc (real)
  GCC Protect enabled:                     no
  GCC march native enabled:                yes
  GCC Profile enabled:                     no
  Position Independent Executable enabled: no
  CFLAGS                                   -g -O2 -march=native -I${srcdir}/../rust/gen/c-headers
  PCAP_CFLAGS
  SECCFLAGS

Files

v5_mem_usage.png (327 KB) v5_mem_usage.png Memory usage on system running 5.0.0, then 5.0.2 Vlad Grigorescu, 02/18/2020 05:38 PM
v4_mem_usage.png (305 KB) v4_mem_usage.png Memory usage on system running 4.1.3 Vlad Grigorescu, 02/18/2020 05:38 PM

Related issues 1 (0 open1 closed)

Related to Suricata - Bug #3455: asan ftp related leaks on the current gitmasterRejectedActions
Actions #1

Updated by Victor Julien about 4 years ago

  • Description updated (diff)
Actions #2

Updated by Victor Julien about 4 years ago

Are you able to compile with something like leak sanitizer? Your compiler looks pretty old, so I'm guessing this could be tricky.

Some other questions:

1. are you using GeoIP rules?
2. are you utilizing the redis output that is compiled in?

Actions #3

Updated by Victor Julien about 4 years ago

Seeing this in my own sensor. Not very helpful output

=================================================================
==25301==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 198 byte(s) in 22 object(s) allocated from:
    #0 0x7efe887d2b50 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb50)
    #1 0x7efe862e62b4 in pcre_get_substring (/lib/x86_64-linux-gnu/libpcre.so.3+0x2c2b4)

SUMMARY: AddressSanitizer: 198 byte(s) leaked in 22 allocation(s).

EDIT: fixed in #3566, will be backported

Actions #4

Updated by Victor Julien about 4 years ago

  • Related to Bug #3455: asan ftp related leaks on the current gitmaster added
Actions #5

Updated by Vlad Grigorescu about 4 years ago

Apologies for the delay; I was away.

The current system is running on CentOS 7.7, so the compiler is indeed rather outdated. We have plans to upgrade to 8, but that's likely several weeks away. I could explore using Software Collections on CentOS 7 to install a newer GCC.

To answer your specific questions:

  1. No GeoIP rules
  2. No redis output yet (just plain old eve.json, http.log, stats.log, suricata.log, and tls.log)

Sounds like the ball is in my court until I can get some more details.

Actions #6

Updated by Vlad Grigorescu almost 4 years ago

I'm not sure we'll be able to upgrade soon. We tried upgrading to CentOS 8, but that doesn't support the RAID controller we have in these systems.

We're looking at ordering replacement systems.

Actions #7

Updated by Andreas Herz about 2 years ago

  • Status changed from New to Closed

Hi, we're closing this issue since there have been no further responses.
If you think this issue is still relevant, try to test it again with the
most recent version of suricata and reopen the issue. If you want to
improve the bug report please take a look at
https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Reporting_Bugs

Actions

Also available in: Atom PDF