Bug #3732
closedfilemagic logging resulting in performance hit
Description
Testing out latest stable (5.0.3) and git - specific corner case exposes a bottle neck with file magic logging.
In a specific high speed pcap replay that is entirely and only http/dns traffic with big number of unique sessions - enabling file magic logging triggers the issue - big drop percentage when the system is actually not busy at all.
What is observed during the runs :
- no memcaps hits
- CPUS are at 15-20%
- no rules loaded (on purpose) , just http and fileino logging enabled
- perf top shows no bottle neck or pegged CPUs in certain functions
- htop shows no CPUs pegged
- top shows no specific threads being pegged or being a bottle neck
Drops hit 50% as soon as the AFP v3 buffers get filled
The configs and pcaps will be shared privately privately
Updated by Peter Manev over 4 years ago
Eric has done a patch, preliminary testing seems to show very good results.
Updated by Victor Julien over 4 years ago
- Status changed from New to In Review
- Assignee set to Eric Leblond
- Target version set to 6.0.0beta1
Updated by Peter Manev over 4 years ago
The patch is good in my tests.
Off note: It seems before the fix the side effect of that was - some big mem usage in libhtp
https://redmine.openinfosecfoundation.org/issues/3735
Updated by Victor Julien over 4 years ago
- Status changed from In Review to Closed
Updated by Victor Julien over 1 year ago
- Related to Feature #5894: file: file classification keyword added