Suricata

As discussed on the forum (https://forum.suricata.io/t/feature-idea-compile-per-group-mpm-hyperscan-databases-in-parallel-during-rule-load-reload/6556), Victor suggested opening a ticket.

PROBLEM

During rule load and reload, building the MPM (Hyperscan) databases for all signature group heads (SGH) runs in a single thread. On a multi-core box the rest of the CPU stays idle while this phase dominates the total reload time, especially after every feed update when many groups change and the sgh-mpm-cache cannot be reused.

MEASUREMENTS

OPNsense 26.7.8, Suricata 8, Intel Core Ultra 7 155H (22 hardware threads), about 120k rules incl. ET Telemetry and abuse.ch.
Columns: parse phase / "rule group set" until "rule reload complete" / total

- Reload after scheduled rule update, no MPM cache: 45 s / 2:13 min / 2:58 min
- Reload after rule update, sgh-mpm-caching: yes (73 of 158 cache files rebuilt): 35 s / 1:42 min / 2:17 min
- Reload without rule changes, MPM cache warm: 31 s / 9 s / 40 s

During the long phase exactly one thread (suricata{suricata}) is at about 99 % for the whole time, the other 21 hardware threads are idle (sampled every 2 s).

IDEA

The MPM databases of the different signature groups are independent of each other. Compiling them with a small worker pool, e.g. limited by a config option such as detect.mpm-compile-threads (default 1 = current behaviour), should reduce this phase roughly by the number of available cores. The existing sgh-mpm-cache already helps for unchanged groups; parallel compilation would help for the groups that do change after each feed update (ThreatFox/URLhaus etc. update several times a day).

RELATED

- #4223 (rule loading takes too much time in certain cases)
- #6376 (IP-only load time, fixed in 8.0)
- OPNsense side, enabling sgh-mpm-caching in the template: https://github.com/opnsense/core/issues/11171