Project

General

Profile

Actions

Bug #5287

open

(Maybe) issues in FTP decoder, Suricata stop analyzing traffic

Added by Odin Jenseg 5 months ago. Updated 3 months ago.

Status:
New
Priority:
High
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

We have observed on several of our sensors that Suricata has stopped analyzing traffic, kernel packets goes to zero, but the sensor still receives packets.

Issues observed on:

  • Centos 7 and Centos Stream 8
  • Suricata 5.0.7 and 6.0.4 (with custom patches to SMB and a proprietary app-layer)
  • Intel and Netronome NICs
  • AF-packet v3
  • With and without XDP
  • No hardware bypass

We started to look into whether any stats continued to update after the kernel packets decreased to zero. One of the stats that was identified was app_layer.expectations, which is also increased by the FTP decoder.

<see image graph>

Blue line: Kernel packets
Yellow line: app_layer.expectations

We are not sure if the increase and update on expectations is just a symptom of something else with the FTP decoder, but we have not observed the issues after we disabled the FTP decoder.

On Suricata 5.0.7, we observed an drastic increase in the memory used by Suricata, which with our resources never really stopped (128 GB - 256 GB RAM). This made us assume some sort of memory leak, but we did not observe the same degree of increase for Suricata 6.0.4 when the same issues appeared, and the increase stopped at a "moderate" amount of some tens of gigabytes. We thought it could be related to #5204, and tried to update our config for ippair. It seems this had no impact on the issue, as we still observed this type of behavior.

Observations from a perf report:

We have a perf report from a sensor that has entered the "kernel packets stuck at zero" state. The image below shows what is observed for two workers.

<see perf image>


Files

image2022-4-25_12-47-6.png (28.7 KB) image2022-4-25_12-47-6.png Perf report image Odin Jenseg, 04/25/2022 01:27 PM
image2022-4-25_12-26-31.png (165 KB) image2022-4-25_12-26-31.png stats graph Odin Jenseg, 04/25/2022 01:27 PM
Actions #1

Updated by Odin Jenseg 3 months ago

Hi,

Have anyone been able to look into this, or observed similar issue? Or if there are any changes related to the FTP decoder that could have fixed this issue.

Actions

Also available in: Atom PDF