Suricata

Not sure where's the best place to keep WIP results, but since this is private for now, I'll share them here.

I've made temporary changes to the code, to:
- dns_log_json_[query/answer]: log tx_id in DNS logs in places where either it wasn't logged or it was first decremented before being logged.
- Alert event: create it adding tx_id that comes from what we track as log_id
- Alert header: add the tx_id used from the PacketAlert info to the output
- dns-json: create event headers using the tx_id that is passed to the function (originally, we don't add this info to this event header)

I have also added 2 more rules:

alert dns any any -> any any (msg:"DNS oisf client"; content:"oisf"; flow:to_client; sid:2; rev:1;)
alert dns any any -> any any (msg:"DNS suricata server"; content:"suricata"; flow:to_server; sid:3; rev:1;) 

Debugging with dns-tcp-multi.pcap:

Suricata only starts to append rules to the AlertQueue after parsing packet 12 - something that I think is related to the tx_id tracking - it seems that the engine only catches up with a packet where it could match the rule with what the packet has around the transactions 4 or 5 - which would explain why we seem to miss the alert for the first transaction, and seemingly alert on wrong data, too - but could be wrong.

The DNS transactions logged don't seem to be necessarily the ones that triggered the alerts. It also doesn't seem to align with the tx_id that is saved in the PacketAlert - at least for some cases?

If I keep only rules sid: 2 and sid:3 (eve-rules-2.-3.json), we only see two alerts - but will log query and answer from what I think should be transactions 5 and 6 (query and answer to suricata.org). When we look at the tx_id logged for the alert for sid:3, from the PacketAlert and from the engine tracking, we see that it's 4. Maybe this could work if we tracked all txs as starting with index 0, but as we use the State's tx_id for many functions that control tx_id handling and tracking - and that one starts from 1, things get messed up, I think. It gets more confusing to me for sid:2, as the oisf.net query tx_id seems to indicate it was detected after the suricata.org one. I must investigate more, I guess.

I'm attaching the resulting eve log (s), including eve-with-more-tx-ids-et-rules.json.

I'll share soon a drawing of what I understand is happening while the engine processes dns-tcp-multi.pcap along with what I thought should be seen at each step - at least in terms of tx_id tracking.

I think it should be possible for us to simplify the tx_id tracking to avoid app-layer having to increment or decrement tx_id info when sending or receiving it from the engine.

TODO: for all cases where we use the tx_id, where are we getting it from, and is it incremented or decremented before we save/log/return?

One attempt to represent what I saw happening for the DNS flow for pcap dns-tcp-multi, with the three rules indicated above, running GDB.
I will bring whatever findings to discuss with Victor tomorrow, and see if any of that makes sense, to know what should be the next step.
Haven't gotten to run this with simulate-ips, yet.

1. if we trigger raw inspection data for TCP app-protos, the engine properly detects the alerts from stream in IDS mode;
2. the false positives happen in IPS mode because of the stream nature of the rule. So the alert is actually due to previous traffic on the stream matching;
3. we'll thus address this by triggering raw stream inspection. I also think I should go back to finishing the frame's documentation from a user perspective, and plan to submit a lightning talk to SuriCon to discuss the types of rules and impacts - such as what we saw with this issue.