Optimization #7860
openwrite_merged high memory usage when not quiet
Description
Hello,
While debugging high memory usage issues in suricata-update I noticed that the write_merged method uses quite a lot of memory when the quiet flag is not set. Below is the output of gnu-time executing suricata-update, one time with suricata-update 1.3.4 from Debian 13 Trixie stock, and one time with the lines of code in the args.quiet if-statement commented:
Stock
# /usr/bin/time --verbose suricata-update Command being timed: "suricata-update" User time (seconds): 36.40 System time (seconds): 3.24 Percent of CPU this job got: 85% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:46.37 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 1190508 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 258 Minor (reclaiming a frame) page faults: 317985 Voluntary context switches: 485 Involuntary context switches: 2426 Swaps: 0 File system inputs: 272232 File system outputs: 162192 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0
Edited
# /usr/bin/time --verbose suricata-update Command being timed: "suricata-update" User time (seconds): 19.57 System time (seconds): 1.73 Percent of CPU this job got: 98% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:21.57 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 761452 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 74 Minor (reclaiming a frame) page faults: 166861 Voluntary context switches: 244 Involuntary context switches: 999 Swaps: 0 File system inputs: 29000 File system outputs: 157800 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0
The biggest difference in my eyes here is the "Maximum resident set size" metric, which dropped from almost 1.2GB to ~761MB. This seems to match my observations from watching top while suricata-update is running. Where the stock version uses ~31% of the memory in this machine (4GB of RAM) and the edited one ~20%.
The memory usage by this step is obviously influenced by how many rules are parsed by suricata-update and the default options that distributions set for suricata-update (Debian doesn't seem to set any in suricata-update.service). The memory usage for an (IMO) not super critical metric is quite big though.
One attempt to reduce the memory usage by changing the added, removed and modified variables to integers (as opposed to lists) didn't seem to yield any significant improvements unfortunately, although it probably wouldn't hurt to do that anyway.