Project

General

Profile

Actions

Bug #3732

closed

filemagic logging resulting in performance hit

Added by Peter Manev almost 4 years ago. Updated almost 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

Testing out latest stable (5.0.3) and git - specific corner case exposes a bottle neck with file magic logging.

In a specific high speed pcap replay that is entirely and only http/dns traffic with big number of unique sessions - enabling file magic logging triggers the issue - big drop percentage when the system is actually not busy at all.
What is observed during the runs :

  • no memcaps hits
  • CPUS are at 15-20%
  • no rules loaded (on purpose) , just http and fileino logging enabled
  • perf top shows no bottle neck or pegged CPUs in certain functions
  • htop shows no CPUs pegged
  • top shows no specific threads being pegged or being a bottle neck

Drops hit 50% as soon as the AFP v3 buffers get filled
The configs and pcaps will be shared privately privately


Related issues 1 (1 open0 closed)

Related to Suricata - Feature #5894: file: file classification keywordFeedbackVictor JulienActions
Actions #1

Updated by Peter Manev almost 4 years ago

Eric has done a patch, preliminary testing seems to show very good results.

Actions #2

Updated by Victor Julien almost 4 years ago

  • Status changed from New to In Review
  • Assignee set to Eric Leblond
  • Target version set to 6.0.0beta1
Actions #3

Updated by Peter Manev almost 4 years ago

The patch is good in my tests.

Off note: It seems before the fix the side effect of that was - some big mem usage in libhtp
https://redmine.openinfosecfoundation.org/issues/3735

Actions #4

Updated by Victor Julien almost 4 years ago

  • Status changed from In Review to Closed
Actions #5

Updated by Victor Julien about 1 year ago

  • Related to Feature #5894: file: file classification keyword added
Actions

Also available in: Atom PDF