Project

General

Profile

Actions

Bug #3480

open

EVE JSON - Incorrect Packet Logged

Added by Eoin Miller almost 5 years ago. Updated 6 months ago.

Status:
New
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

In reviewing the contents of the EVE JSON output, we noticed that the value for the key named "packet" was not the value of the packet that matched the rule. In this case, it is the value of the next packet in the flow from the client.

Packet 4:

0000   2c 0b e9 48 aa 7a ac 87 a3 32 ed 91 08 00 45 00   ,..H.z...2....E.
0010   00 e7 00 00 40 00 40 06 00 00 0a 0b 03 17 36 f0   ....@.@.......6.
0020   aa cb e5 b7 00 50 60 d9 81 c4 1c 0f 9b 7e 80 18   .....P`......~..
0030   08 04 ef b6 00 00 01 01 08 0a 41 81 b9 60 0e fb   ..........A..`..
0040   2d f1 50 4f 53 54 20 2f 54 68 69 73 49 73 41 54   -.POST /ThisIsAT
0050   65 73 74 20 48 54 54 50 2f 31 2e 31 0d 0a 48 6f   est HTTP/1.1..Ho
0060   73 74 3a 20 72 61 70 69 64 37 2e 63 6f 6d 0d 0a   st: rapid7.com..
0070   55 73 65 72 2d 41 67 65 6e 74 3a 20 63 75 72 6c   User-Agent: curl
0080   2f 37 2e 35 34 2e 30 0d 0a 41 63 63 65 70 74 3a   /7.54.0..Accept:
0090   20 2a 2f 2a 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79    */*..Content-Ty
00a0   70 65 3a 61 70 70 6c 69 63 61 74 69 6f 6e 2f 78   pe:application/x
00b0   2d 77 77 77 2d 66 6f 72 6d 2d 75 72 6c 65 6e 63   -www-form-urlenc
00c0   6f 64 65 64 0d 0a 43 6f 6e 74 65 6e 74 2d 4c 65   oded..Content-Le
00d0   6e 67 74 68 3a 20 32 35 0d 0a 0d 0a 57 68 6f 44   ngth: 25....WhoD
00e0   6f 59 6f 75 53 75 70 70 6f 72 74 3d 55 70 47 61   oYouSupport=UpGa
00f0   6c 77 61 79 21                                    lway!

Packet 7:

0000   2c 0b e9 48 aa 7a ac 87 a3 32 ed 91 08 00 45 00   ,..H.z...2....E.
0010   00 34 00 00 40 00 40 06 00 00 0a 0b 03 17 36 f0   .4..@.@.......6.
0020   aa cb e5 b7 00 50 60 d9 82 77 1c 0f 9d c9 80 10   .....P`..w......
0030   07 fa ef 03 00 00 01 01 08 0a 41 81 b9 81 0e fb   ..........A.....
0040   2d f5                                             -.

eve.json log's value for the key named "packet" matches packet 7 from pcap file:

jq -r '.packet' eve.json  | base64 -D | hexdump -C
00000000  2c 0b e9 48 aa 7a ac 87  a3 32 ed 91 08 00 45 00  |,..H.z...2....E.|
00000010  00 34 00 00 40 00 40 06  00 00 0a 0b 03 17 36 f0  |.4..@.@.......6.|
00000020  aa cb e5 b7 00 50 60 d9  82 77 1c 0f 9d c9 80 10  |.....P`..w......|
00000030  07 fa ef 03 00 00 01 01  08 0a 41 81 b9 81 0e fb  |..........A.....|
00000040  2d f5                                             |-.|
00000042

It is worth noting that the correct packet appears to be in the unified2 output (also attached to ticket).


Files

UpGalway.pcap (1.59 KB) UpGalway.pcap Eoin Miller, 02/14/2020 08:36 PM
eve.json (1.32 KB) eve.json Eoin Miller, 02/14/2020 08:36 PM
unified2.out (329 Bytes) unified2.out Eoin Miller, 02/14/2020 08:37 PM

Related issues 5 (4 open1 closed)

Related to Suricata - Feature #1380: JSON and Unified2 output "payload" does not contain full (or real in the case of Unified2) packets for sessionNewOISF Dev02/09/2015Actions
Related to Suricata - Bug #2429: TCP-session and wrong alert timestamp RejectedOISF DevActions
Related to Suricata - Bug #2069: logging: payload may not represent traffic the generated alert (eve and unified2)AssignedJason IshActions
Related to Suricata - Feature #2281: tcp stream: simpler IDS handling of overlap evasionsAssignedVictor JulienActions
Related to Suricata - Documentation #5690: userguide: document the differences between IPS and IDS modeNewOISF DevActions
Actions #1

Updated by Peter Manev almost 5 years ago

I think it seems the packet logged should be packet 5 (not 7) as it is the ACK of the POST (packet 4). In IDS we act on acknowledged data.
Can you share the rule and try the latest master to see if there is any difference?

Actions #2

Updated by Eoin Miller almost 5 years ago

Peter Manev wrote:

I think it seems the packet logged should be packet 5 (not 7) as it is the ACK of the POST (packet 4). In IDS we act on acknowledged data.
Can you share the rule and try the latest master to see if there is any difference?

Sure thing. Packet #4 is the one that matches the rule (and ends up in the unified2 output attached as unified2.out) but the packet that is logged in the EVE output is Packet #7:

Here is the rule we were using while testing:

alert http any any -> any any (msg:"RAPID7 TEST TestHTTPPost - Up Galway"; flow:established,to_server; content:"POST"; http_method; content:"WhoDoYouSupport=UpGalway!"; http_client_body; sid:7777777; rev:1;)

I'll try it against current tomorrow to see if the behavior is the same tomorrow.

Actions #3

Updated by Eoin Miller almost 5 years ago

Verified same behavior with release 5.0.2.

Actions #4

Updated by Jason Ish almost 5 years ago

The packet data found in unified2 is more like the "payload" option of eve - in unified2 the packet is rebuilt based on the payload, its not the actual packet. The main difference in eve payload being that we don't fake out the full packet like we do in unified2.

Actions #5

Updated by Eoin Miller almost 5 years ago

Jason Ish wrote:

The packet data found in unified2 is more like the "payload" option of eve - in unified2 the packet is rebuilt based on the payload, its not the actual packet. The main difference in eve payload being that we don't fake out the full packet like we do in unified2.

Alrighty, this raises a few questions from our end:

  1. So are people supposed to be using '.payload' instead of '.packet'?
  2. Should the value of '.packet' not be the packet that caused the match?

From what I've been told by our team members, '.payload' seems to be the first up to 4k of the stream and doesn't appear to include packet headers. With unified2 on its way out, making sure that what is logged with EVE gives analysts what they need to triage alerting will become more important.

Actions #6

Updated by Jason Ish almost 5 years ago

So they payload should be the same as seen in unified2, minus the headers, as can pretty much be seen when you base64 decode the payload in the eve.json you provided. The rest of the details, like the addresses should be in the event itself. Is that not enough to triage on?

I'm just trying to point out that there is no less data with the loss of unified2, it just in a bit of a different format (which may not be idea in its own right, or not what people are used, etc).

Actions #7

Updated by Eoin Miller almost 5 years ago

Jason Ish wrote in #note-6:

So they payload should be the same as seen in unified2, minus the headers, as can pretty much be seen when you base64 decode the payload in the eve.json you provided. The rest of the details, like the addresses should be in the event itself. Is that not enough to triage on?

I'm just trying to point out that there is no less data with the loss of unified2, it just in a bit of a different format (which may not be idea in its own right, or not what people are used, etc).

Did some more digging, I think the disconnect on our end was:

  1. Thinking that '.packet' was the packet that caused the alert, in full.
  2. '.payload' only outputs up to the first 4k by default.

By increasing the value for outputs -> eve-log -> types -> alert -> payload-buffer-size, we do see the same output as we would inside of unified2.

It may be helpful to:
  1. Increase default value for payload-buffer-size from 4k to 64k so that the full packet is in the eve output by default
  2. Update the suricata.yaml comments for packet:

From:

            # packet: yes              # enable dumping of packet (without stream segments)

To:
            # packet: yes              # enable dumping of packet headers (without stream segments/data/payload)

It does seem a bit weird still that the value for '.packet' doesn't match the headers for the packet that contained the data payload that matched the rule.

Actions #8

Updated by Philippe Antoine about 3 years ago

  • Affected Versions 6.0.3 added

I have been investigating this.

It seems to me the real bug is that we run detection not as early as we can (and should)
On pcap UpGalway.pcap, the signature alert http any any -> any any (msg:"RAPID7 TEST TestHTTPPost - Up Galway"; http.method; content:"POST"; sid:7777777;) produces an alert at packet 7.
That is why we see this packet 7 in eve.json

I think it seems the packet logged should be packet 5 (not 7) as it is the ACK of the POST (packet 4). In IDS we act on acknowledged data.

Suricata indeed does the app-layer parsing on packet 5.
But then, when it runs DetectRun with only the packet direction, so we do not match this rule because the rule is to server and the packet 5 is to client.
Packet 7 is the first packet in the direction to server after we parsed the data.
That is why alert happens on packet 7.

One workaround is to use stream.inline=true to get alert at packet 4

What could be the fix ?
- Could we take the rules group in the opposing direction when it is TCP and not inline ?
No because packet rules such as with tcp.hdr have the right direction
(We need also to keep app-layer protocols supported by both TCP and UDP working)
- Could we run DetectRun twice on both directions ?
Maybe, but it looks expensive in terms of performance, and kind of annihilating the benefit of the rules grouping...
Merging TCP rules groups for both directions would also diminish the benefit of the rules grouping I guess...
- Could we change the rules group where we put these kind of rules (over TCP, app-layer, when not inline) ?
Yes but it is not enough :
+ DetectBufferMpmRegistery have a direction as well which needs to be consistent with one in the sig group
+ We also need to revert flow_flags in DetectRunTx so that we get the right transaction progress (the one in the right direction) and so that DetectEngineAppInspectionEngine's direction is consistent as well
+ Other things that I did not notice yet...

This seems to be require deep design change...
Thoughts ?

Actions #9

Updated by Jason Ish over 2 years ago

  • Related to Feature #1380: JSON and Unified2 output "payload" does not contain full (or real in the case of Unified2) packets for session added
Actions #10

Updated by Jason Ish over 2 years ago

  • Related to Bug #2429: TCP-session and wrong alert timestamp added
Actions #11

Updated by Jason Ish over 2 years ago

  • Related to Bug #2069: logging: payload may not represent traffic the generated alert (eve and unified2) added
Actions #12

Updated by Jason Ish over 2 years ago

Just linking up some related issues as I think its worth a discussion and/or documentation.

- Related to #1380, as I think it relates to the discussion of payload vs packet.
- Related to #2429 as this shows the discrepancy in time between the triggering packet and the timestamp in the alert which is also seen in EVE.

From my own observations, in IDS mode:
- The packet logged is the next packet in the stream. You won't find the alerting data in this packet and often the payload in this packet is empty.
- The timestamp of the alert does not align with the packet that contains the triggering data, but instead some packet later.
- The payload does contain the triggering data.

In IPS mode:
- The packet logged DOES contain the alert triggering data. If the data spans multiple packets, the packet logged is the completing packet (which I think is expected behaviour). So you may not see all the data leading to an alert to trigger, but thats OK, at least the packet logged correlates to the input.
- The alert timestamp does match the packet timestamp.
- They payload is similar to that in IDS mode, you can see the re-assembled payload leading to the alert to trigger.

Note that there are occurrences where the payload logged may not include the triggering data, as seen in issue #2069. However, I believe this affects IDS mode only, and not IPS mode as detection is run sooner in IPS mode.

Ideally what is logged for alerts should be the same between IDS and IPS mode. Perhaps even going as far as removing packet from the default config as it has no value in IDS deployments. Even them it would be nice if the timestamp was correct for correlations against full packet capture systems.

Actions #13

Updated by Jason Ish about 2 years ago

  • Related to Feature #2281: tcp stream: simpler IDS handling of overlap evasions added
Actions #14

Updated by Jason Ish about 2 years ago

  • Related to Documentation #5690: userguide: document the differences between IPS and IDS mode added
Actions #15

Updated by Philippe Antoine over 1 year ago

  • Assignee set to OISF Dev
Actions #16

Updated by Jason Ish 10 months ago

  • Affected Versions git master added
Actions #17

Updated by Philippe Antoine 6 months ago

  • Target version set to TBD
Actions

Also available in: Atom PDF