Project

General

Profile

Actions

Bug #1526

open

Malformed encoded base64 packet in json logs

Added by Jay MJ about 7 years ago. Updated 7 months ago.

Status:
New
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

I have been testing erspan support with git 2.1, rev e583de0 and rev 834c366. I noticed that the encoded base64 packets in the eve log appear to be corrupt, missing the IP header, and containing mostly apparent invalid Ethernet header information. Have not had a chance to test these specific versions without erspan yet, but may in the future (hoping someone else can test this).

Decoded packet fields contain these fields (will provide pcap if requested):
Ether - dst,src,proto type (always strange, unknown values), Raw - load

Looking at the first 50 bytes of the decoded packet field, it does not appear to be an erspan packet (verified also by cutting erspan header length, which completely malforms the pcap).

I also output pcaps from suricata to file (<200 MB each), which does properly work. All pcaps contain erspan header information, and appear normal.

Test box specs:
Running on virtualized esx host
ArchLinux 4.1.4-1 kernel, x86_64
8-core, 16 GB ram
erspan inbound vnic Intel 82545EM


Files

Sample mangled base64 decoded.pcap (110 Bytes) Sample mangled base64 decoded.pcap Sample decoded pcap Jay MJ, 09/16/2015 08:23 AM
Actions #1

Updated by Bakul Khanna about 7 years ago

I see a very similar issue with Suricata 2.1 Beta 4.

While running Modbus traffic via a pcap file, with EVE logging on, sometimes I see a corrupted base64 encoded "packet" field. At other times the base64 encoded "packet" field is perfectly fine.

------------------------------------------------------------------------------------
Following is an example of a EVE log with a corrupted base64 encoded "packet" field:

2015-08-26T10:58:16.364-07:00 Suricata[16810]: {"timestamp":"2015-08-26T10:58:16.359538-0700","flow_id":28633488,"event_type":"alert","src_ip":"192.168.1.1","src_port":47762,"dest_ip":"192.168.1.2","dest_port":502,"proto":"TCP","alert":{"action":"allowed","gid":1,"signature_id":2250003,"rev":1,"signature":"SURICATA Modbus invalid Length","category":"","severity":3,"tx_id":0},
"payload":"0MrzAgAAAAAAEAAABwQAAAFvAAAEAf8HAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=","payload_printable":".................o.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................","stream":1,"packet":"AODtEcfoACOuLYlXCABFAAAoCudFAAAobJXAqAEGwKjAqAEBwKgBArqSAfb4WTHgBQrFLQAAAAAAAAAA"}
"payload_printable":".................o.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................",
"stream":1,"packet":"AODtEcfoACOuLYlXCABFAAAoCudFAAAobJXAqAEGwKjAqAEBwKgBArqSAfb4WTHgBQrFLQAAAAAAAAAA"}

I decoded the "packet" as follows (Note how the ip addresses decoded are different from what is mentioned in the log above):
•Frame 1 (60 bytes on wire, 60 bytes captured)
•Ethernet II, Src: Dell_2d:89:57 (00:23:ae:2d:89:57), Dst: Silicom_11:c7:e8 (00:e0:ed:11:c7:e8)
•Internet Protocol, Src: 192.168.1.6 (192.168.1.6), Dst: 192.168.192.168 (192.168.192.168)
•Data (20 bytes)
•0000 01 01 c0 a8 01 02 ba 92 01 f6 f8 59 31 e0 05 0a ...........Y1...
•0010 c5 2d 00 00 .-..

------------------------------------------------------------------------------------
Following is an example of a EVE log with a perfectly fine base64 encoded "packet" field:

2015-08-26T10:58:16.365-07:00 Suricata[16810]: {"timestamp":"2015-08-26T10:58:16.359538-0700","flow_id":28633488,"in_iface":"eth0","event_type":"alert","src_ip":"192.168.1.2","src_port":502,"dest_ip":"192.168.1.1","dest_port":47762,"proto":"TCP","alert":{"action":"allowed","gid":1,"signature_id":2250003,"rev":1,"signature":"SURICATA Modbus invalid Length","category":"","severity":3,"tx_id":0},
"payload":"",
"stream":1,"packet":"AODtEcfoACOuLYlXCABFAAAoCudAAIAGbJXAqAECwKgBAQH2upIFCsUt+Fkx4FAUAAB7ggAAAAAAAAAA"}

I decoded the "packet" as follows (Note how the ip addresses decoded are identical to what is mentioned in the log above):
•Frame 1 (60 bytes on wire, 60 bytes captured)
•Ethernet II, Src: Dell_2d:89:57 (00:23:ae:2d:89:57), Dst: Silicom_11:c7:e8 (00:e0:ed:11:c7:e8)
•Internet Protocol, Src: 192.168.1.2 (192.168.1.2), Dst: 192.168.1.1 (192.168.1.1)
•Transmission Control Protocol, Src Port: asa-appl-proto (502), Dst Port: 47762 (47762), Seq: 84591917, Ack: 4166595040, Len: 0

Actions #2

Updated by Jay MJ about 7 years ago

Provided more log (json alert, alert debug, scapy output) information here: http://pastebin.com/H2pwTUXj

Also attached sample base64 decoded pcap.

It looks like the encoding isn't properly done or something about it changed in suricata somewhere. I've tried a few versions of scapy to decode the packet field, and get the same results.

For me, it's always malformed, there haven't been any proper packet fields since testing erspan.

Actions #3

Updated by Jay MJ almost 7 years ago

It appears md5sum hashes are also incorrect.

Actions #4

Updated by Jay MJ almost 7 years ago

Confirmed that the issues below are only related to erspan traffic. Tested with regular rspan and the issues do not occur.

Issues
  • Incorrect MD5sum (file magic appears to be correct)
  • Malformed packet field in eve log

OS information
Linux stplidsn02 4.2.3-1-ARCH #1 SMP PREEMPT Sat Oct 3 18:52:50 CEST 2015 x86_64 GNU/Linux
Suricata git ~ This is Suricata version 2.1dev (rev dcbbda5)

If developers need any further information or testing, please let me know.

Actions #5

Updated by Victor Julien almost 7 years ago

Can you reproduce the issues with a pcap recording of your erspan traffic as well?

Actions #6

Updated by Jay MJ almost 7 years ago

Jay MJ wrote:

Confirmed that the issues below are only related to erspan traffic. Tested with regular rspan and the issues do not occur.

Issues
  • Incorrect MD5sum (file magic appears to be correct)
  • Malformed packet field in eve log

OS information
Linux stplidsn02 4.2.3-1-ARCH #1 SMP PREEMPT Sat Oct 3 18:52:50 CEST 2015 x86_64 GNU/Linux
Suricata git ~ This is Suricata version 2.1dev (rev dcbbda5)

If developers need any further information or testing, please let me know.

Apologies, please disregard the hashing issue, that is a separate issue, and appears to be present across spans. I had made some configuration changes that probably borked that.

Actions #7

Updated by Jay MJ almost 7 years ago

Victor Julien wrote:

Can you reproduce the issues with a pcap recording of your erspan traffic as well?

The pcaps suricata saves and tcpdump as well are the same, and OK. It appears to be how suricata parses the pcap.

Actions #8

Updated by Victor Julien almost 7 years ago

What capture method are you using? If you're not using regular libpcap (-i on commandline) can you try that? Wondering if possibly there is something about our afpacket/pfring support that is related to this.

Actions #9

Updated by Jay MJ almost 7 years ago

Victor Julien wrote:

What capture method are you using? If you're not using regular libpcap (-i on commandline) can you try that? Wondering if possibly there is something about our afpacket/pfring support that is related to this.

I was using af-packet, tried doing libpcap, same results happen.

I think it has to do with the erspan component of suricata. Plain rspan itself works fine, but once I revert to erspan, the packet alert field becomes malformed. What I think is happening is the packet alert field parsing is not addressing the proceeding 50 byte erspan header. Then, subsequently generated pcap gets improperly offset and encoded into the alert packet field of suricata.

Actions #10

Updated by Jay MJ almost 7 years ago

Victor Julien wrote:

What capture method are you using? If you're not using regular libpcap (-i on commandline) can you try that? Wondering if possibly there is something about our afpacket/pfring support that is related to this.

Also tried pfring, the results are the same.

Actions #11

Updated by Jay MJ over 6 years ago

Looking at this some more, appears perhaps the GRE header itself could be a factor when packet field gets encoded for eve log? I'll do some more testing in time.

Actions #12

Updated by Jay MJ over 6 years ago

It looks like a couple of things are happening comparing to rspan and erspan and pf_ring on both. The alert packet field is encoding in base64 but I believe it does so by looking at the original encoded GRE/ERSPAN packet, so everything is malformed.

I also noticed that the bpf filter often doesn't work with this encoded traffic most of the time. In fact suricata won't pick up any traffic (unless bpf is something simple like tcp protocol). If you would like another ticket for this let me know.

So, at this point I would recommend:
  • Mapping the packet field to properly decoded GRE/ESPAN packet rather than raw
  • Apply bpf filters to properly decoded GRE/ERSPAN packet; not sure what it's using - perhaps also raw encoded original?

Not sure if there is much demand for erspan support, seems like I'm the only one using it. I also looked into rcdcap to handle decoding pre suricata with a tunnel interface, however the software is older, and I have a pending issue with the developer of that project.

Actions #13

Updated by Andreas Herz about 6 years ago

  • Assignee set to Anonymous
  • Target version set to TBD
Actions #14

Updated by Jay MJ almost 6 years ago

Not sure if someone started working on this (I see some activity in git), but it appears that the md5sum is working now and correct (but sha1/256 are still incorrect). I am testing 3.2beta1, one erspan comes in stripped of erspan header and then passed to suricata, other erspan is fed straight to suricata. Strangely both md5 hashes match and are correct, however sha1 and sha256 have both different values and are not the correct hash of the file. This may be another bug, as my test shows even without erspan header the sha1/256 hashes are incorrect.

Also, the alert packet field decoded at initial look appears to be correct now, although I would like to do more research for a couple of days before calling that good.

If any additional testing needs to be done, don't hesitate to let me know.

Actions #15

Updated by Jay MJ over 5 years ago

The hash issue was fixed in 3.2.1, Bug #2004.

Still seeing malformed packet fields though (only with erspan traffic).

Actions #16

Updated by Jay MJ about 5 years ago

Jay MJ wrote:

The hash issue was fixed in 3.2.1, Bug #2004.

Still seeing malformed packet fields though (only with erspan traffic).

Still seeing the issue with malformed pcaps, packet / payload fields in eve logs with suricata 4.
Testing with rcdcap (virtual tap) which strips the erspan headers and everything works fine. Happy to do any testing / provide more info if needed.

Actions #17

Updated by Andreas Herz over 3 years ago

  • Assignee set to Community Ticket
Actions #18

Updated by Jay MJ about 3 years ago

Jay MJ wrote:

The hash issue was fixed in 3.2.1, Bug #2004.

Still seeing malformed packet fields though (only with erspan traffic).

Updating the proper ticket this time!

I have noticed that the only issue remaining in v 4.1.4 is the packet field in the eve log, which doesn't seem to be valid. The encoded packet field does not work when imported into scapy, or decoded and converted to a pcap file. Everything else appears to work properly (encoded payload, printed payload, recorded pcaps) with or without the erspan header.

Actions #19

Updated by Andreas Herz about 3 years ago

  • Assignee changed from Community Ticket to OISF Dev

Thanks for the update, we need to check that last missing piece of the puzzle

Actions #20

Updated by Jason Ish 7 months ago

Are you able to provide an input pcap as well as a rule that shows this issue?

Actions

Also available in: Atom PDF