Suricata

I think it seems the packet logged should be packet 5 (not 7) as it is the ACK of the POST (packet 4). In IDS we act on acknowledged data.
Can you share the rule and try the latest master to see if there is any difference?

Peter Manev wrote:

I think it seems the packet logged should be packet 5 (not 7) as it is the ACK of the POST (packet 4). In IDS we act on acknowledged data.
Can you share the rule and try the latest master to see if there is any difference?

Sure thing. Packet #4 is the one that matches the rule (and ends up in the unified2 output attached as unified2.out) but the packet that is logged in the EVE output is Packet #7:

Here is the rule we were using while testing:

alert http any any -> any any (msg:"RAPID7 TEST TestHTTPPost - Up Galway"; flow:established,to_server; content:"POST"; http_method; content:"WhoDoYouSupport=UpGalway!"; http_client_body; sid:7777777; rev:1;)

I'll try it against current tomorrow to see if the behavior is the same tomorrow.

Verified same behavior with release 5.0.2.

The packet data found in unified2 is more like the "payload" option of eve - in unified2 the packet is rebuilt based on the payload, its not the actual packet. The main difference in eve payload being that we don't fake out the full packet like we do in unified2.

Jason Ish wrote:

The packet data found in unified2 is more like the "payload" option of eve - in unified2 the packet is rebuilt based on the payload, its not the actual packet. The main difference in eve payload being that we don't fake out the full packet like we do in unified2.

Alrighty, this raises a few questions from our end:

1. So are people supposed to be using '.payload' instead of '.packet'?
2. Should the value of '.packet' not be the packet that caused the match?

From what I've been told by our team members, '.payload' seems to be the first up to 4k of the stream and doesn't appear to include packet headers. With unified2 on its way out, making sure that what is logged with EVE gives analysts what they need to triage alerting will become more important.

So they payload should be the same as seen in unified2, minus the headers, as can pretty much be seen when you base64 decode the payload in the eve.json you provided. The rest of the details, like the addresses should be in the event itself. Is that not enough to triage on?

I'm just trying to point out that there is no less data with the loss of unified2, it just in a bit of a different format (which may not be idea in its own right, or not what people are used, etc).

Jason Ish wrote in #note-6:

So they payload should be the same as seen in unified2, minus the headers, as can pretty much be seen when you base64 decode the payload in the eve.json you provided. The rest of the details, like the addresses should be in the event itself. Is that not enough to triage on?

I'm just trying to point out that there is no less data with the loss of unified2, it just in a bit of a different format (which may not be idea in its own right, or not what people are used, etc).

Did some more digging, I think the disconnect on our end was:

1. Thinking that '.packet' was the packet that caused the alert, in full.
2. '.payload' only outputs up to the first 4k by default.

By increasing the value for outputs -> eve-log -> types -> alert -> payload-buffer-size, we do see the same output as we would inside of unified2.

1. Increase default value for payload-buffer-size from 4k to 64k so that the full packet is in the eve output by default
2. Update the suricata.yaml comments for packet:

From:

            # packet: yes              # enable dumping of packet (without stream segments)

            # packet: yes              # enable dumping of packet headers (without stream segments/data/payload)

It does seem a bit weird still that the value for '.packet' doesn't match the headers for the packet that contained the data payload that matched the rule.

Just linking up some related issues as I think its worth a discussion and/or documentation.

- Related to #1380, as I think it relates to the discussion of payload vs packet.
- Related to #2429 as this shows the discrepancy in time between the triggering packet and the timestamp in the alert which is also seen in EVE.

From my own observations, in IDS mode:
- The packet logged is the next packet in the stream. You won't find the alerting data in this packet and often the payload in this packet is empty.
- The timestamp of the alert does not align with the packet that contains the triggering data, but instead some packet later.
- The payload does contain the triggering data.

In IPS mode:
- The packet logged DOES contain the alert triggering data. If the data spans multiple packets, the packet logged is the completing packet (which I think is expected behaviour). So you may not see all the data leading to an alert to trigger, but thats OK, at least the packet logged correlates to the input.
- The alert timestamp does match the packet timestamp.
- They payload is similar to that in IDS mode, you can see the re-assembled payload leading to the alert to trigger.

Note that there are occurrences where the payload logged may not include the triggering data, as seen in issue #2069. However, I believe this affects IDS mode only, and not IPS mode as detection is run sooner in IPS mode.

Ideally what is logged for alerts should be the same between IDS and IPS mode. Perhaps even going as far as removing packet from the default config as it has no value in IDS deployments. Even them it would be nice if the timestamp was correct for correlations against full packet capture systems.