Project

General

Profile

Optimization #2803

test phase should use less memory

Added by Victor Julien 11 days ago. Updated 5 days ago.

Status:
Closed
Priority:
Normal
Target version:
Effort:
Difficulty:

Description

While suricata-update runs it's test phase, its memory use remains high. This seems unnecessary, as all the rule parsing & sorting is already done. The only result of the test is pass or fail, no complicated rule actions should be done anymore.

History

#1

Updated by Jason Ish 11 days ago

2 things come to mind to immediately solve this. Set some of the dict's that are storing all the state to None and see if they get garbage collected. Or break the big main() up into smaller functions, which is more likely to garbage collect stuff as they get reference counted to 0 on function return, plus main() could probably benefit some breaking it up a bit.

#2

Updated by Victor Julien 11 days ago

I should add that I only tested with python2. The way I observed this is simple: htop showed the mem use of suricata-update while the test command (a suricata -T variant) was running. It can probably be simulated by simply having a test script that sleeps for some time.

#3

Updated by Jason Ish 5 days ago

  • Status changed from Assigned to Closed
  • Target version changed from 1.1.0 to 1.0.4

Fixed by setting some collection types to None before running the test. https://github.com/OISF/suricata-update/commit/5cea9cf4f29fca00ede5c0882f2f9356415f3aba

This is probably about the best we can do without some redesign of suricata-update internals.

Also available in: Atom PDF