Project

General

Profile

Actions

Bug #4499

open

Sudden and enormous memory leak

Added by Gianni Tedesco almost 3 years ago. Updated 11 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
Affected Versions:
Effort:
Difficulty:
Label:

Description

Hi,

We have a large installed base of suricata 6.0.2. We're seeing a memory leak that can consume unbounded amounts of memory (even hundreds of gigabytes over a few days). The leak appears suddenly in bursts, not gradually over time. This is affecting between 15% and 20% of all our suricata instances.

We cannot find anything in common between the leaky instances other than that they all see the following protocols: TLS, DNS, HTTP and DCE-RPC.

Attached a graph showing various telemetry memuse stats over time alongside resident set size.

We were seeing the issue also in 6.0.1, and we think we were also seeing it with 6.0.0 judging by some OOM kernel panics we saw...


Files

leak.png (151 KB) leak.png Gianni Tedesco, 05/20/2021 06:42 AM
build.txt (4.13 KB) build.txt suricata --build Gianni Tedesco, 05/20/2021 07:05 AM
config.txt (15.8 KB) config.txt suricata --dump-config Gianni Tedesco, 05/20/2021 07:05 AM
malloc-trim.png (171 KB) malloc-trim.png malloc_trim(0) only trims a few hundred MB Gianni Tedesco, 05/24/2021 07:51 AM
Actions #2

Updated by Gianni Tedesco almost 3 years ago

We've ruled out DCE-RPC by disabling it and confirming that the leak still happens.

Actions #3

Updated by Gianni Tedesco almost 3 years ago

We've ruled out memory fragmentation by using gdb to attach to suricata and calling malloc_trim(0).

Actions #4

Updated by Victor Julien over 2 years ago

Can you share stats.log entries to see if they hold any clues?

Actions #5

Updated by Victor Julien 11 months ago

@Gianni Tedesco are you still seeing this issue?

Actions

Also available in: Atom PDF