Bug #3488
closedMemory leak in 5.0.2
Description
We were seeing a bad memory leak in Suricata 5.0.0, and upgraded to 5.0.2. The situation is significantly better, but there's still a memory leak as compared with 4.1.3.
I'm attaching screenshots of memory usage over 1 week from two systems looking at very similar traffic. For the system running Suricata v5, you can see when we upgraded from 5.0.0 to 5.0.2 on 2/13. However, the memory use is slowly creeping up.
suricata --build-info from both systems follow:
This is Suricata version 5.0.2 RELEASE
Features: PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LUA HAVE_LUAJIT HAVE_LIBJANSSON TLS MAGIC RUST
SIMD support: SSE_3
Atomic intrinsics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 4.8.5 20150623 (Red Hat 4.8.5-39), C version 199901
compiled with _FORTIFY_SOURCE=0
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.32, linked against LibHTP v0.5.32
Suricata Configuration:
AF_PACKET support: yes
eBPF support: no
XDP support: no
PF_RING support: no
NFQueue support: no
NFLOG support: no
IPFW support: no
Netmap support: no
DAG enabled: no
Napatech enabled: no
WinDivert enabled: no
Unix socket enabled: yes
Detection enabled: yes
Libmagic support: yes
libnss support: yes
libnspr support: yes
libjansson support: yes
hiredis support: yes
hiredis async with libevent: no
Prelude support: no
PCRE jit: yes
LUA support: yes, through luajit
libluajit: yes
GeoIP2 support: yes
Non-bundled htp: no
Old barnyard2 support: no
Hyperscan support: no
Libnet support: yes
liblz4 support: yes
Rust support: yes
Rust strict mode: no
Rust compiler path: /usr/bin/rustc
Rust compiler version: rustc 1.38.0
Cargo path: /usr/bin/cargo
Cargo version: cargo 1.38.0
Cargo vendor: yes
Python support: yes
Python path: /usr/bin/python2.7
Python distutils yes
Python yaml yes
Install suricatactl: yes
Install suricatasc: yes
Install suricata-update: yes
Profiling enabled: no
Profiling locks enabled: no
Development settings:
Coccinelle / spatch: no
Unit tests enabled: no
Debug output enabled: no
Debug validation enabled: no
Generic build parameters:
Installation prefix: /usr/local
Configuration directory: /usr/local/etc/suricata/
Log directory: /usr/local/var/log/suricata/
--prefix /usr/local
--sysconfdir /usr/local/etc
--localstatedir /usr/local/var
--datarootdir /usr/local/share
Host: x86_64-pc-linux-gnu
Compiler: gcc (exec name) / gcc (real)
GCC Protect enabled: no
GCC march native enabled: yes
GCC Profile enabled: no
Position Independent Executable enabled: no
CFLAGS -g -O2 -march=native -I${srcdir}/../rust/gen/c-headers
PCAP_CFLAGS
SECCFLAGS
--------------------------------------
This is Suricata version 4.1.3 RELEASE
Features: PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LUA HAVE_LUAJIT HAVE_LIBJANSSON TLS MAGIC RUST
SIMD support: SSE_4_2 SSE_4_1 SSE_3
Atomic intrisics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 4.8.5 20150623 (Red Hat 4.8.5-36), C version 199901
compiled with _FORTIFY_SOURCE=0
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.30, linked against LibHTP v0.5.30
Suricata Configuration:
AF_PACKET support: yes
eBPF support: no
XDP support: no
PF_RING support: no
NFQueue support: no
NFLOG support: no
IPFW support: no
Netmap support: no
DAG enabled: no
Napatech enabled: no
WinDivert enabled: no
Unix socket enabled: yes
Detection enabled: yes
Libmagic support: yes
libnss support: yes
libnspr support: yes
libjansson support: yes
liblzma support: yes
hiredis support: yes
hiredis async with libevent: no
Prelude support: no
PCRE jit: yes
LUA support: yes, through luajit
libluajit: yes
libgeoip: yes
Non-bundled htp: no
Old barnyard2 support: no
Hyperscan support: no
Libnet support: yes
liblz4 support: yes
Rust support: yes (default)
Rust strict mode: no
Rust debug mode: no
Rust compiler: rustc 1.36.0
Rust cargo: cargo 1.36.0
Install suricatasc: yes
Install suricata-update: yes
Profiling enabled: no
Profiling locks enabled: no
Development settings:
Coccinelle / spatch: no
Unit tests enabled: no
Debug output enabled: no
Debug validation enabled: no
Generic build parameters:
Installation prefix: /usr/local/security/suricata/builds/suricata_4.1.3
Configuration directory: /usr/local/security/suricata/builds/suricata_4.1.3/etc/suricata/
Log directory: /usr/local/security/suricata/builds/suricata_4.1.3/var/log/suricata/
--prefix /usr/local/security/suricata/builds/suricata_4.1.3
--sysconfdir /usr/local/security/suricata/builds/suricata_4.1.3/etc
--localstatedir /usr/local/security/suricata/builds/suricata_4.1.3/var
--datarootdir /usr/local/security/suricata/builds/suricata_4.1.3/share
Host: x86_64-pc-linux-gnu
Compiler: gcc (exec name) / gcc (real)
GCC Protect enabled: no
GCC march native enabled: yes
GCC Profile enabled: no
Position Independent Executable enabled: no
CFLAGS -g -O2 -march=native -I${srcdir}/../rust/gen/c-headers
PCAP_CFLAGS
SECCFLAGS
Files
Updated by Victor Julien over 4 years ago
Are you able to compile with something like leak sanitizer? Your compiler looks pretty old, so I'm guessing this could be tricky.
Some other questions:
1. are you using GeoIP rules?
2. are you utilizing the redis output that is compiled in?
Updated by Victor Julien over 4 years ago
Seeing this in my own sensor. Not very helpful output
=================================================================
==25301==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 198 byte(s) in 22 object(s) allocated from:
#0 0x7efe887d2b50 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb50)
#1 0x7efe862e62b4 in pcre_get_substring (/lib/x86_64-linux-gnu/libpcre.so.3+0x2c2b4)
SUMMARY: AddressSanitizer: 198 byte(s) leaked in 22 allocation(s).
EDIT: fixed in #3566, will be backported
Updated by Victor Julien over 4 years ago
- Related to Bug #3455: asan ftp related leaks on the current gitmaster added
Updated by Vlad Grigorescu over 4 years ago
Apologies for the delay; I was away.
The current system is running on CentOS 7.7, so the compiler is indeed rather outdated. We have plans to upgrade to 8, but that's likely several weeks away. I could explore using Software Collections on CentOS 7 to install a newer GCC.
To answer your specific questions:
- No GeoIP rules
- No redis output yet (just plain old eve.json, http.log, stats.log, suricata.log, and tls.log)
Sounds like the ball is in my court until I can get some more details.
Updated by Vlad Grigorescu over 4 years ago
I'm not sure we'll be able to upgrade soon. We tried upgrading to CentOS 8, but that doesn't support the RAID controller we have in these systems.
We're looking at ordering replacement systems.
Updated by Andreas Herz almost 3 years ago
- Status changed from New to Closed
Hi, we're closing this issue since there have been no further responses.
If you think this issue is still relevant, try to test it again with the
most recent version of suricata and reopen the issue. If you want to
improve the bug report please take a look at
https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Reporting_Bugs