Bug #7824: hyperscan: caching results in segfault with link time optimization (-flto=auto, etc) - Suricata - Open Information Security Foundation

Actions

Copy link

Bug #7824

closed

hyperscan: caching results in segfault with link time optimization (-flto=auto, etc)

Added by A. IOOSS about 1 month ago. Updated 24 days ago.

Status:

Closed

Priority:

Normal

Assignee:

Lukas Sismis

Target version:

8.0.1

Affected Versions:

8.0.0

Effort:

Difficulty:

Label:

Description

Ubuntu 24.04LTS, with https://launchpad.net/~oisf/+archive/ubuntu/suricata-stable PPA

`apt policy suricata` shows:

 *** 1:8.0.0-0ubuntu2 500
    500 https://ppa.launchpadcontent.net/oisf/suricata-stable/ubuntu noble/main amd64 Packages

It's a stock Ubuntu VPS running Docker services, stock Suricata configuration, but the systemd unit is patched with the following to run Suricata in nfq IPS mode:

/usr/bin/suricata -q 0 -k none --set security.lua.allow-rules=yes --set outputs.1.eve-log.enabled=no --set nfq.mode=repeat --set nfq.repeat-mark=1 --set nfq.repeat-mask=1 -c /etc/suricata/suricata.yaml --pidfile /run/suricata.pid

and the following rule is added:

iptables -I FORWARD -m mark ! --mark 1/1 -j NFQUEUE --queue-num 0 --queue-bypass

In /var/log/suricata/suricata.log, the following lines appears multiple time:

<snip>
[216562 - W#15] 2025-07-19 18:46:57 Error: suricata: stacktrace:sig 11:PrefilterAddSidsResize+0x000000e2;avx2_hs_reset_and_expand_stream+0x000db76b;avx2_hs_reset_and_expand_stream+0x000d7d15;avx2_hs_database_info+0x0000af8e;avx2_hs_reset_and_expand_stream+0x00009d98;avx2_hs_scan+0x000005ed;SCHSSearch+0x00000068;DetectEngineInspectFiledata+0x00000357;DetectRunPrefilterTx+0x00000075;DetectXbitsRegister+0x00001f2d;Detect+0x00000137;FlowClearMemory+0x000003a6;TmThreadsSlotVarRun+0x00000048;SCTmThreadsSlotPacketLoopFinish+0x00000376;pthread_condattr_setpshared+0x00000684;__clone+0x0000024c
<snip>
[226927 - W#09] 2025-07-19 18:52:57 Error: suricata: stacktrace:sig 11:PrefilterAddSidsResize+0x000000e2;avx2_hs_reset_and_expand_stream+0x000db76b;avx2_hs_reset_and_expand_stream+0x000d7d15;avx2_hs_database_info+0x0000af8e;avx2_hs_reset_and_expand_stream+0x00009d98;avx2_hs_scan+0x000005ed;SCHSSearch+0x00000068;DetectEngineInspectFiledata+0x00000357;DetectRunPrefilterTx+0x00000075;DetectXbitsRegister+0x00001f2d;Detect+0x00000137;FlowClearMemory+0x000003a6;TmThreadsSlotVarRun+0x00000048;SCTmThreadsSlotPacketLoopFinish+0x00000376;pthread_condattr_setpshared+0x00000684;__clone+0x0000024c
<snip>

Attached you may find a coredump. An initial look seems to indicate that the crash happens because `pat` is -1 in `SCHSWatchEvent` function.

As this bug was discovered while playing ENOWARS9 attack-defense CTF event, on a provided temporary VPS, I do not have access to this machine anymore.

Files

_usr_bin_suricata.108.crash (16.3 MB) _usr_bin_suricata.108.crash

A. IOOSS, 07/20/2025 10:01 AM

Actions

Copy link

Updated by Jason Ish about 1 month ago

Private changed from No to Yes

Actions

Copy link

Updated by A. IOOSS about 1 month ago · Edited

The rule engine is buggy when using hyperscan. This can be reproduced on a fresh Ubuntu 24.04 install, with the same Suricata package, and same firewall configuration.

The Suricata rules are:

alert ip any any -> any any (msg: "Found toto"; flow:to_server; content: "toto";  sid: 4001;)
alert ip any any -> any any (msg: "Found GET"; flow:to_server; content: "GET";  sid: 4002;)

If the second rule (4002) is commented, then toto is correctly detected.
If both rules are uncommented, toto is not longer detected.

The problem disappears if Suricata is run with these options:

--set mpm-algo=ac --set spm-algo=bm

Actions

Copy link

Updated by Jason Ish about 1 month ago

Subject changed from Suricata 8.0.0 segfault in IPS mode to hyperscan: caching results in segfault
Assignee changed from OISF Dev to Lukas Sismis
Private changed from Yes to No

I ran into this as well. There are 2 workarounds I've found so far:

The first is to disable caching:

  sgh-mpm-caching: no

But it also appears related to the `-flto=auto` CFLAG, probably with other `-flto` options, but removing `-flto=auto` during the builds appears to solve it as well, at least in the RPM. I'm guessing the Debian package might be doing something similar, but I can't confirm at the moment.

Actions

Copy link

Updated by Jason Ish about 1 month ago

Subject changed from hyperscan: caching results in segfault to hyperscan: caching results in segfault with link time optimization (-flto=auto, etc)
Target version changed from TBD to 8.0.1