Suricata » Suricata-Update

Hello,

While debugging high memory usage issues in suricata-update I noticed that the write_merged method uses quite a lot of memory when the quiet flag is not set. Below is the output of gnu-time executing suricata-update, one time with suricata-update 1.3.4 from Debian 13 Trixie stock, and one time with the lines of code in the args.quiet if-statement commented:

Stock

# /usr/bin/time --verbose suricata-update
Command being timed: "suricata-update" 
User time (seconds): 36.40
System time (seconds): 3.24
Percent of CPU this job got: 85%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:46.37
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 1190508
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 258
Minor (reclaiming a frame) page faults: 317985
Voluntary context switches: 485
Involuntary context switches: 2426
Swaps: 0
File system inputs: 272232
File system outputs: 162192
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

Edited

# /usr/bin/time --verbose suricata-update
Command being timed: "suricata-update" 
User time (seconds): 19.57
System time (seconds): 1.73
Percent of CPU this job got: 98%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:21.57
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 761452
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 74
Minor (reclaiming a frame) page faults: 166861
Voluntary context switches: 244
Involuntary context switches: 999
Swaps: 0
File system inputs: 29000
File system outputs: 157800
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

The biggest difference in my eyes here is the "Maximum resident set size" metric, which dropped from almost 1.2GB to ~761MB. This seems to match my observations from watching top while suricata-update is running. Where the stock version uses ~31% of the memory in this machine (4GB of RAM) and the edited one ~20%.

The memory usage by this step is obviously influenced by how many rules are parsed by suricata-update and the default options that distributions set for suricata-update (Debian doesn't seem to set any in suricata-update.service). The memory usage for an (IMO) not super critical metric is quite big though.

One attempt to reduce the memory usage by changing the added, removed and modified variables to integers (as opposed to lists) didn't seem to yield any significant improvements unfortunately, although it probably wouldn't hurt to do that anyway.