Support #5287
open(Maybe) issues in FTP decoder, Suricata stop analyzing traffic
Description
We have observed on several of our sensors that Suricata has stopped analyzing traffic, kernel packets goes to zero, but the sensor still receives packets.
Issues observed on:
- Centos 7 and Centos Stream 8
- Suricata 5.0.7 and 6.0.4 (with custom patches to SMB and a proprietary app-layer)
- Intel and Netronome NICs
- AF-packet v3
- With and without XDP
- No hardware bypass
We started to look into whether any stats continued to update after the kernel packets decreased to zero. One of the stats that was identified was app_layer.expectations, which is also increased by the FTP decoder.
<see image graph>
Blue line: Kernel packets
Yellow line: app_layer.expectations
We are not sure if the increase and update on expectations is just a symptom of something else with the FTP decoder, but we have not observed the issues after we disabled the FTP decoder.
On Suricata 5.0.7, we observed an drastic increase in the memory used by Suricata, which with our resources never really stopped (128 GB - 256 GB RAM). This made us assume some sort of memory leak, but we did not observe the same degree of increase for Suricata 6.0.4 when the same issues appeared, and the increase stopped at a "moderate" amount of some tens of gigabytes. We thought it could be related to #5204, and tried to update our config for ippair. It seems this had no impact on the issue, as we still observed this type of behavior.
Observations from a perf report:
We have a perf report from a sensor that has entered the "kernel packets stuck at zero" state. The image below shows what is observed for two workers.
<see perf image>
Files
Updated by Odin Jenseg about 2 years ago
Hi,
Have anyone been able to look into this, or observed similar issue? Or if there are any changes related to the FTP decoder that could have fixed this issue.
Updated by Victor Julien about 2 months ago
- Tracker changed from Bug to Support
- Priority changed from High to Normal
- Target version deleted (
TBD)
@Odin Jenseg Is this still an issue in 7.0.6?