Weird handling of IKEv2 flows when alerts happen
IKEv2 parser in its current state makes it possible to cause a slight denial-of-service condition to single suricata capture thread when alert is hit.
alert ikev2 any any -> ikev2.server.ip.here any (msg:"SURICATA IKEv2"; sid:2; rev:1;)
My initial analysis follows.
It seems that the IKEv2 transaction vector for the flow in rust/src/ikev2/ikev2.rs (IKEV2State struct) gets filled with flows. Now let's say I fill it with 10000 IKEv2 exchanges that will amount to 10000 iterations (by get_tx_by_id) per new IKEv2 exchange and on top of that it seems that with every new IKEv2 exchange I make will cause get_tx_by_id be called 10 times. (so I get 100000 iterations per IKEv2 exchange which eats up fair bit of CPU time)
The flows are cleaned up as expected when flows timeout (which will clean up the transaction vector for that flow). Normally this would work but in this case the nature of IKE makes matters worse. Any new exchange will keep the flow alive (due to being srcip:500 <-> dstip:500) and this prevents the transaction vector from being cleaned up.
When alert does not happen, the connection is almost immediately cleaned up in AppLayerParserTransactionsCleanup (src/app-layer-parser.c) but the alert rule I described does cause us to hit goto next due to detect_flags_ts & APP_LAYER_TX_INSPECTED_FLAG being true. In non-alert case the transaction is freed by StateTransactionFree call before the next: goto label in AppLayerParserTransactionsCleanup function.
Tested on 4.1.4 and latest git head.
I believe this to be the same issue as https://redmine.openinfosecfoundation.org/issues/3877.