Suricata

Commit d8ddef4c (https://github.com/OISF/suricata/commit/d8ddef4c) ("detect: delay tx cleanup in some edge case") added a condition in AppLayerParserTransactionsCleanup that delays transaction cleanup until FLOW_SGH_TOCLIENT is set. However, when a pass rule matches on a UDP flow, FLOW_ACTION_PASS causes DetectFlow() to skip all subsequent packets in both directions, so FLOW_SGH_TOCLIENT is never set and APP_LAYER_TX_INSPECTED_TC is never marked. This means transactions are never freed for the lifetime of the flow. On long-lived UDP flows (e.g. continuous SNMP polling), this results in unbounded memory growth as app-layer transactions accumulate indefinitely. The memory is eventually freed when the flow times out, but for continuously active flows that never happens. A minimal repro is attached: a single pass udp 10.0.0.0/8 any -> any any rule with looped SNMP traffic on one flow demonstrates linear heap growth via the SNMP add_pdu_info OID vectors in massif. Additionally, this likely happens with other UDP-based App Layer protocols (although I have not tested myself).

The repro was done using the af-packet runmode. I have included a pcap (replayed with tcpreplay), as well as all relevant config files for reproing this issue, along with a massif report that was generated while running valgrind on the Suricata binary (both the report file and the output from ms_print.