Suricata

We also saw, on live deployments, that the flow.timeout_inuse counter is between 30-70% of the overall flows. So might seem to be another correlation that would hint that some flows don't timeout as before

Is it possible to post: build-info and pef top, top -H -p suripid when that starts to happen?
Also there is no midstream pickups in the config enabled right?

midstream pickup is not enabled.

Both runs you see are the same traffic (via trex), same config it's just the different versions of Suricata.

5.0 stable memuse as expected with the stable traffic, with 6.0 it is increasing and after some hours will hit the memcap.

I am not sure if ti is related, but noticed the following for 5.x from the info provided

compiled with LibHTP v0.5.33, linked against LibHTP v0.5.37

Also rust is not the same version. 1.34 for 5.x and 1.42 for 6.0.
Wondering if it has any effect - just something worth of "equalizing" to remove it as potential cause.

good catch, we did a rebuild with the same versions that are used for 6.0 but no change, 5.0 still looking good and 6.0 bad. So we can rule that out as well.

We could at least narrow it down to this commit set:

last known good commit https://github.com/OISF/suricata/commit/be57f6d3cc74f3bb1315439218c24a928cae7c48
first known bad https://github.com/OISF/suricata/commit/999af4f62a234d78f682c2b842f434410a61f56f (is just documentation)

So in between are several Flow related commits from that I see in https://github.com/OISF/suricata/pull/5284 as well. Which would make sense since the relevant sections have been touched/changed. But the change is rather complex so might take some time to find out what causes the big memory increase/"leak".

So either the whole rework or a specific commit seems to be the reason for that.

A deeper dive revealed:

last known good commit https://github.com/OISF/suricata/commit/48605f48674b1478a7dfcb03d67fa2d92eff4f51#diff-ae20d432e75e77294238b0a3633016e6efaa39c1b759d2a1d0b18e135bed7eac
first bad one https://github.com/OISF/suricata/commit/b3599507f4eb891841417575587d690ea13fe6c0#diff-ae20d432e75e77294238b0a3633016e6efaa39c1b759d2a1d0b18e135bed7eac

in between the commit https://github.com/OISF/suricata/commit/2fd7c87f2271b14ba8c1ed4246c04fbe0457b1cf#diff-ae20d432e75e77294238b0a3633016e6efaa39c1b759d2a1d0b18e135bed7eac introcuded another regression that lead to a very low tcp reassembly memuse but high flow use and much more events and flows on a T-rex test setup which also resulted in much higher drops, but nearly 0 tcp reassembly gaps.

I think this issue is related to what I observed in https://redmine.openinfosecfoundation.org/issues/4200. We did observe issues around increasing flows in the eBPF map that was never deleted, and a huge difference between the number of "closed" bypass flows in 5 and 6. Its been a while since I did any testing on 6, but using Suricata 5 did resolve our issue.

Comparing the two outputs between the good and the bad one you can see that the good one has 92% in the SCMallocFunc tree and just 4% for the SCReallocFunc while that increases in the bad one to 42%. This could explain the increase in memory usage if it's not released or at least not released fast enough.
So looking into flow-worker and tm-threads changes will be the next step.

I did a check on all deployments and while I can see the issue everywhere I can also confirm that it's not solely related to `tcp.reassembly_gap` since I have several deployments with the issue that don't have any gaps.
So IMHO most of the correlation are just a sideeffect.

(Possible) Related forum posts:

https://forum.suricata.io/t/kernel-drops-and-optimization-recommendations/1044/5

https://forum.suricata.io/t/accumulation-of-flow-timeout-inuse/880/10 (with nearly the same graphana output)