Bug #6963
openrule-reload: potential memory leak in multiple rule reloads
Description
There is a potential memory leak present in Suricata 6.0.18 and 7.0.4 that is revealed by the memory usage on rule-reloads.
This doesn't need any traffic being forwarded.
To reproduce it, start Suricata with the ET Open ruleset and default/basic settings. You can use a dummy interface instead of an actual interface where traffic is forwarded to.
Once Suricata did start check the memory usage, for example with htop. After 2-3 minutes trigger a rule reload (either via suricatasc or sending USR2 signal).
Observe the memory output and repeat this a few times. You should see that in most cases the memory usage increases during the reload, is reduced a bit in the end but the overall diff between the time before and after the reload is positive.
On a testrun with 6.0.18 I saw the following usage for the Suricata process with the first value being "VIRT" and the second one being "RES" memory value read from htop:
Those are the values for 7.0.4
The PR https://github.com/OISF/suricata/pull/9756 which is linked at https://redmine.openinfosecfoundation.org/issues/6454 doesn't change that issue (I tried a backport of that PR to 7.0.4)
AH Updated by Andreas Herz over 2 years ago
Some more additions, the first bump in memory usage happens with the call of "SigLoadSignatures" from 3271/883 to 3888/1499 in the example, this is expected to be an increase since the rules are loaded in parallel while the current ruleset is still active and will be swapped later.
The second bump happens when "DetectEngineReloadThreads" is triggered, the bump is much smaller, from the 3888/1499 to 3907/1502 so very minor.
With the "DetectEnginePruneFreeList" most is freed again down to "3587/1182" and the new addition with the malloc PR reduces it further down to "3587/964" which makes sense that less RES memory is used when the "malloc_trim" happens. But we still have an overall increase that will go up for each reload and thus overtime the system memory will be exhausted at one point.
AH Updated by Andreas Herz over 2 years ago
Further investigation with different runs of Suricata with the default `glibc`, `jemalloc` and `tcmalloc` showed no diff on the root issue. There is a slight diff on how much memory is used (see table below) but the steady increase is there in all 3 cases.
The additional output described in https://blog.inliniac.net/2014/12/23/profiling-suricata-with-jemalloc/ for jemalloc also showed no actual leak, so it's more a logical "leak"
Also `runmode=single` shows the issue as well.
Memory usage (VIRT and RES) after the start, after the first reload, after the second reload
AH Updated by Andreas Herz about 2 years ago
I had some time to play around with the suggestion from Victor to see if it's related to threshold, classification and/or reference. I tried a ET ruleset where I removed all metadata (except sid/rev/msg) and all thresholds. The "leak" is still present there as well.
PA Updated by Philippe Antoine about 1 year ago
Is it still the case for 8 ?
Can you get a flame graph of allocated memory to see where the leak is happening ?
PA Updated by Philippe Antoine about 1 year ago
- Status changed from New to Feedback