Suricata

SUMMARY
In IPS mode (AF_PACKET copy-mode ips, stream inline), long-lived TLS connections are
silently dropped by the exception policy when the client-to-server direction reaches
exactly 2^32 + 2^31 + stream.reassembly.depth bytes (6,476,005,376 bytes with depth 32mb).

The engine logs a drop with reason "applayer error" on a to_client packet. In that stats
interval app_layer.error.tls.gap, tcp.reassembly_gap and
exception_policy.app_layer.error.drop_flow increase by one, and every later packet of the
flow is dropped with reason "flow drop". The endpoints get no RST/FIN; the sender keeps
retransmitting until it times out.

About 2^31 bytes after the reassembly depth is reached, the engine starts raising stream
events on most packets of the flow (est_packet_out_of_window, est_invalid_ack,
est_pkt_before_last_ack, pkt_spurious_retransmission) but keeps forwarding them. At
2^32 + 2^31 + depth a gap is declared in the to_client direction; the TLS parser does not
accept gaps, returns an error, and exception-policy auto (drop-flow in IPS mode) drops the
flow.

The behaviour is deterministic for a given pcap, happens with an empty ruleset, is
identical in 8.0.7, and shifts exactly with stream.reassembly.depth. Not every long flow is
affected: we have seen 100-250 GB flows pass untouched, and we could not identify what
distinguishes them.

ENVIRONMENT
- Suricata 8.0.5 (Debian trixie-backports 1:8.0.5-1~bpo13+1), LibHTP 8.0.5, Hyperscan 5.4.2
- Also reproduced offline with Suricata 8.0.7 (1:8.0.7-1~bpo13+1)
- Debian 13 (trixie), kernel 6.12.95, NIC Broadcom bnxt_en, offloads disabled
- AF_PACKET IPS, copy-mode ips, 2 interfaces, cluster_flow, 9 worker threads per
  interface, workers runmode
- ~115k rules loaded in production; the issue also reproduces with no rules

RELEVANT CONFIGURATION (at the time of the incident)
exception-policy: auto
stream:
  memcap: 12gb
  memcap-policy: drop-flow
  checksum-validation: yes
  midstream: false          # configured as the invalid value "detect", which evaluates to false
  inline: auto
  reassembly:
    memcap: 6gb
    depth: 32mb
    toserver-chunk-size: 8192
    toclient-chunk-size: 8192
    randomize-chunk-size: yes
app-layer:
  protocols:
    tls:
      enabled: yes          # encryption-handling not set (default: track-only)
      detection-ports:
        dp: 443
flow-timeouts:
  tcp: {new: 60, established: 600, closed: 60, bypassed: 100}

TRAFFIC
Proxmox Backup Server "push" sync jobs: one TCP connection per VM group, TLS 1.3 + HTTP/2
on port 8007. The client uploads at 2-20 MiB/s (also seen at ~400 Mbit/s); the server sends
small responses. Affected connections are cut every time at the same point: over two weeks,
more than 30 connections died at about 6,476,005,376 bytes (within a few MB; our estimate
includes retransmitted bytes) or at that value + 2^32.

EVIDENCE (one connection; client ISN 2252758, server ISN 2772709017; offsets are relative
to the client ISN)
- reassembly_depth_reached (to_server): relative seq 33,553,937
- first est_packet_out_of_window (to_server): relative seq 2,188,168,293 (= 2^31 + 40,684,645)
- first est_invalid_ack (to_client), 50 ms later: ack 2,188,169,741
- drop "applayer error" on a to_client pure ACK (len 64); last client byte forwarded:
  6,474,699,461
- stats interval of the drop: tcp.reassembly_gap +1, app_layer.error.tls.gap +1,
  exception_policy.app_layer.error.drop_flow +1, ips.drop_reason.applayer_error +1;
  tcp.reassembly_memuse ~30 MB (memcap not involved)
- flow record: "action":"drop",
  "exception_policy":[{"target":"app_layer_error","policy":"drop_flow"}], tcp "tc_gap":true
- stream-event counters for that flow: est_packet_out_of_window 1,486,168;
  est_pkt_before_last_ack 1,494,668; pkt_spurious_retransmission 1,494,489;
  est_invalid_ack 396,918; reassembly_seq_gap 1
- captures on both bridge interfaces confirm that after the drop packets keep arriving
  in both directions and none are forwarded

OFFLINE REPRODUCTION
suricata -c suricata.yaml -r flow.pcap --simulate-ips --runmode single \
  --set stream.inline=yes --set threading.set-cpu-affinity=no
- same drop on the same packet (same timestamp) as in production
- same result with an empty ruleset (-S empty.rules) and with 8.0.7
- stream.reassembly.depth from 4mb to 32mb: the onset of the out-of-window events and the
  drop shift by exactly the change in depth; with 40mb/48mb/64mb the drop falls after the
  end of the capture

WORKAROUND
app-layer.protocols.tls.encryption-handling: bypass, together with stream.midstream: true
(without it, locally bypassed flows expire after flow-timeouts.tcp.bypassed and are
dropped as midstream when traffic resumes). Verified in production: a connection that was
always cut at 6.03 GB transferred 8.25 GB and completed.
app-layer.error-policy: ignore also avoids the drop, but the stream events still occur.

EXPECTED BEHAVIOUR
Long-lived flows should not be dropped once they exceed the reassembly depth.

PCAP
We have a ~7 GB pcap (ingress traffic of both directions of the flow) that reproduces the
issue deterministically. It contains internal addresses and encrypted backup data, so we
cannot attach it publicly, but we can share it privately with the developers.