Suricata

The TCP window can be up to 1GiB in size when wscale=14 and window=0xffff. If client and server agree on this, or if an attacker is able to inject packets with these properties to the IDS/IPS, packets within this window will be accepted.

Using just a few packets this scenario can lead to massive memory use:

1. 3whs agrees on wscale=14
2. packet on the right edge of the window comes in, so at almost 1GiB from the ISN, but still in window
3. stream buffer implementation alloc'd a buffer for the entire sequence space, so 1GiB.

To make things worse, if the evil client/server then ACK's this packet, another packet can be sent at an offset of 1GiB and now the memory region will be expanded to 2GiB. This trick can be done on both directions, so it's possible to consume several gigabytes of memory using just a handful of small packets.

Limiting factors:
- stream.reassembly.depth limits this, but is unlimited by default on some protocols like SMB
- stream.reassembly.memcap limits these extreme numbers by default, but its still trivial to consume smaller amounts and exhaust the memcap

The root cause is that the streaming buffer implementation unconditionally aims to create a continues block of reassembled data in memory. This works well is regular and slightly lossy traffic, but not in this case.

Note that #4580 appears to be a related problem with the same root cause, but it happens organically in high speed setups.

Also related is #5703. If pushed enough, the buffer logic will cause a crash. EDIT: This seems not the case. Can't push it beyond 3GiB per direction. ACKs are needed for moving the window forward, but also trigger a slide/trim.