Suricata

Summary¶

With encryption-handling: bypass in IDS mode, a ja3.hash rule that sets a flowbit and a ja3s.hash rule that depends on it via flowbits:isset produce only 1 alert instead of the expected 2. The bypass-triggered detection flush creates pseudo-packets in to_client-first order, causing the dependent rule to evaluate before the flowbit has been set.

The same rules + same PCAP produce the correct 2 alerts with default/track-only, and also with bypass in IPS/inline mode.

Affected Versions¶

• 7.0.12 RELEASE (Linux and Windows)
• 8.0.3 RELEASE (Windows)
• Current main (code paths verified -- logic unchanged, minor API renames only)

Affected Code¶

src/stream-tcp.c, StreamTcpDetectLogFlush() -- pseudo-packet creation order.

src/flow-worker.c, FlowWorkerStreamTCPUpdate() -- flush trigger and pseudo-packet dequeue loop.

Reproduction¶

Replay the attached PCAP in IDS mode with these two rules, once with encryption-handling: bypass and once with encryption-handling: default (track-only on main/8.x). Everything else identical.

Rules:

alert tls any any -> any any (msg:"JA3 match, flowbits set"; flow:established,to_server; ja3_hash; content:"fae0e5d973c96ae1888b99538efa0363"; flowbits:set,JA3_MATCH; sid:1;)

alert tls any any -> any any (msg:"JA3S match, JA3 flowbits check"; flow:established,to_client; ja3s.hash; content:"907bf3ecef1c987c889946b737b43de8"; flowbits:isset,JA3_MATCH; sid:2;)

sid:1 matches a JA3 hash on the Client Hello and sets flowbit JA3_MATCH. sid:2 matches a JA3S hash on the Server Hello and requires that flowbit via flowbits:isset. The intent is to correlate client and server fingerprints within the same TLS session.

PCAP (attached as docs.suricata.io.pcapng): a single TLS 1.3 connection to docs.suricata.io (18 packets).

Expected (both configurations): 2 alerts (sid:1 and sid:2).
Actual with bypass: 1 alert (sid:1 only). sid:2 does not fire.
Actual with default/track-only: 2 alerts (correct).

The JA3S hash is present in the "event_type":"tls" log event for the bypassed flow -- the hash is available, but the rule doesn't alert.

What Happens¶

Step 1 -- Bypass sets the NO_INSPECTION flag¶

When encryption-handling: bypass is active and the TLS parser sees the first Application Data record, it sets APP_LAYER_PARSER_NO_INSPECTION (among others) in app-layer-ssl.c, inside the SSLV3_APPLICATION_PROTOCOL case (around line 2560):

if (ssl_config.encrypt_mode == SSL_CNF_ENC_HANDLE_BYPASS) {
    SCAppLayerParserStateSetFlag(pstate, APP_LAYER_PARSER_NO_REASSEMBLY);
    SCAppLayerParserStateSetFlag(pstate, APP_LAYER_PARSER_NO_INSPECTION);
    SCAppLayerParserStateSetFlag(pstate, APP_LAYER_PARSER_BYPASS_READY);
}

Both JA3 and JA3S hashes are already computed at this point -- this is not a data-availability problem.

Step 2 -- The NO_INSPECTION flag triggers a detection flush¶

On the next packet, FlowWorkerStreamTCPUpdate() in flow-worker.c (around line 384) detects the flag and calls StreamTcpDetectLogFlush():

bool setting_nopayload =
    p->flow->alparser &&
    SCAppLayerParserStateIssetFlag(p->flow->alparser, APP_LAYER_PARSER_NO_INSPECTION) &&
    !(p->flags & PKT_NOPAYLOAD_INSPECTION);

if (FlowChangeProto(p->flow) || setting_nopayload) {
    StreamTcpDetectLogFlush(tv, fw->stream_thread, p->flow, p, &fw->pq);
    ...
}

The flush creates two pseudo-packets to give detection one last chance to inspect the TLS transaction before the flow is bypassed.

Step 3 -- The flush creates to_client-first pseudo-packets¶

StreamTcpDetectLogFlush() in stream-tcp.c (around line 7100):

bool ts = PKT_IS_TOSERVER(p) ? true : false;
ts ^= StreamTcpInlineMode();
StreamTcpPseudoPacketCreateDetectLogFlush(tv, stt, p, ssn, pq, ts^0);  // first
StreamTcpPseudoPacketCreateDetectLogFlush(tv, stt, p, ssn, pq, ts^1);  // second

The dir parameter is 0=ts, 1=tc per the inner function's doc. In IDS mode, StreamTcpInlineMode() returns 0, so the XOR leaves ts unchanged. The triggering packet is to_server (the first Application Data), so:

• ts = 1, XOR with 0 → ts = 1
• First pseudo: ts^0 = 1 → to_client
• Second pseudo: ts^1 = 0 → to_server

(Note: the comment above this function says "In IDS mode, create first in packet dir, 2nd in opposing / In IPS mode, do the reverse." This is backwards relative to the code.)

Step 4 -- The to_client pseudo runs first; flowbit check fails¶

The pseudo-packets are enqueued into a FIFO queue, then dequeued and processed in flow-worker.c (around line 400):

Packet *x;
while ((x = PacketDequeueNoLock(&fw->pq))) {
    Detect(tv, x, det_ctx);
    OutputLoggerLog(tv, x, fw->output_thread);
    ...
}

Because the to_client pseudo is dequeued first, sid:2 (flowbits:isset,JA3_MATCH) evaluates before sid:1 has run. The flowbit hasn't been set yet, so sid:2 fails. Then sid:1 runs on the to_server pseudo and sets the flowbit -- but it's too late.

Step 5 -- Detection progress tracking prevents re-evaluation¶

After the flush, the original triggering packet also runs through detection. But by this point, detect_progress has been advanced (in detect-engine-prefilter.c, via DetectRunPrefilterTx, around line 135):

if (tx->detect_progress > engine->ctx.tx_min_progress) {
    SCLogDebug("tx already marked progress as beyond engine: %u > %u",
            tx->detect_progress, engine->ctx.tx_min_progress);
    goto next;
}

sid:2 never gets another chance to evaluate with the flowbit set. (In Suricata 7, this mechanism uses a bitmask called prefilter_flags instead of detect_progress, but the effect is the same.)

Why default/track-only works¶

With default (renamed to track-only on main), APP_LAYER_PARSER_NO_INSPECTION is not set (only NO_INSPECTION_PAYLOAD is set), so no flush occurs. Rules evaluate on separate real packets in natural order: sid:1 matches on a to_server packet first, then sid:2 evaluates on a later to_client packet and finds the flowbit already set.

Why bypass + IPS/inline mode works¶

In IPS/inline mode, detection runs on each packet as it arrives (before forwarding). sid:1 matches on a real to_server packet (the ClientHello) before bypass activates. By the time the flush creates pseudo-packets, the flowbit is already set, so sid:2 succeeds regardless of pseudo-packet order.

(The XOR in StreamTcpDetectLogFlush also flips the pseudo-packet order to to_server-first in inline mode, but this is moot since the flowbit is already set from the real packet.)

Summary table¶

bypass  + IDS         → 1 alert (FAIL)  — flush creates to_client-first pseudo-packets, flowbit not set in time
bypass  + IPS/inline  → 2 alerts (pass) — sid:1 matches on real packet before bypass activates
default/track-only    → 2 alerts (pass) — no flush, rules evaluate on separate real packets

Impact¶

• Any pair of TLS rules using flowbits across to_server/to_client directions will fail when encryption-handling: bypass is used in IDS mode.
• The documentation states that encryption-handling controls behavior "after the handshake", implying handshake-related inspection (including JA3/JA3S) should complete before bypass takes effect. In practice, the bypass-triggered flush collapses both directions into back-to-back pseudo-packets where the to_client direction runs first, breaking the flowbit dependency.

References¶

• Suricata docs -- encrypted traffic
• stream-tcp.c -- StreamTcpDetectLogFlush on main
• flow-worker.c -- FlowWorkerStreamTCPUpdate on main