Suricata

I see a very similar issue with Suricata 2.1 Beta 4.

While running Modbus traffic via a pcap file, with EVE logging on, sometimes I see a corrupted base64 encoded "packet" field. At other times the base64 encoded "packet" field is perfectly fine.

------------------------------------------------------------------------------------
Following is an example of a EVE log with a corrupted base64 encoded "packet" field:

2015-08-26T10:58:16.364-07:00 Suricata[16810]: {"timestamp":"2015-08-26T10:58:16.359538-0700","flow_id":28633488,"event_type":"alert","src_ip":"192.168.1.1","src_port":47762,"dest_ip":"192.168.1.2","dest_port":502,"proto":"TCP","alert":{"action":"allowed","gid":1,"signature_id":2250003,"rev":1,"signature":"SURICATA Modbus invalid Length","category":"","severity":3,"tx_id":0},
"payload":"0MrzAgAAAAAAEAAABwQAAAFvAAAEAf8HAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=","payload_printable":".................o.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................","stream":1,"packet":"AODtEcfoACOuLYlXCABFAAAoCudFAAAobJXAqAEGwKjAqAEBwKgBArqSAfb4WTHgBQrFLQAAAAAAAAAA"}
"payload_printable":".................o.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................",
"stream":1,"packet":"AODtEcfoACOuLYlXCABFAAAoCudFAAAobJXAqAEGwKjAqAEBwKgBArqSAfb4WTHgBQrFLQAAAAAAAAAA"}

I decoded the "packet" as follows (Note how the ip addresses decoded are different from what is mentioned in the log above):
•Frame 1 (60 bytes on wire, 60 bytes captured)
•Ethernet II, Src: Dell_2d:89:57 (00:23:ae:2d:89:57), Dst: Silicom_11:c7:e8 (00:e0:ed:11:c7:e8)
•Internet Protocol, Src: 192.168.1.6 (192.168.1.6), Dst: 192.168.192.168 (192.168.192.168)
•Data (20 bytes)
•0000 01 01 c0 a8 01 02 ba 92 01 f6 f8 59 31 e0 05 0a ...........Y1...
•0010 c5 2d 00 00 .-..

------------------------------------------------------------------------------------
Following is an example of a EVE log with a perfectly fine base64 encoded "packet" field:

2015-08-26T10:58:16.365-07:00 Suricata[16810]: {"timestamp":"2015-08-26T10:58:16.359538-0700","flow_id":28633488,"in_iface":"eth0","event_type":"alert","src_ip":"192.168.1.2","src_port":502,"dest_ip":"192.168.1.1","dest_port":47762,"proto":"TCP","alert":{"action":"allowed","gid":1,"signature_id":2250003,"rev":1,"signature":"SURICATA Modbus invalid Length","category":"","severity":3,"tx_id":0},
"payload":"",
"stream":1,"packet":"AODtEcfoACOuLYlXCABFAAAoCudAAIAGbJXAqAECwKgBAQH2upIFCsUt+Fkx4FAUAAB7ggAAAAAAAAAA"}

I decoded the "packet" as follows (Note how the ip addresses decoded are identical to what is mentioned in the log above):
•Frame 1 (60 bytes on wire, 60 bytes captured)
•Ethernet II, Src: Dell_2d:89:57 (00:23:ae:2d:89:57), Dst: Silicom_11:c7:e8 (00:e0:ed:11:c7:e8)
•Internet Protocol, Src: 192.168.1.2 (192.168.1.2), Dst: 192.168.1.1 (192.168.1.1)
•Transmission Control Protocol, Src Port: asa-appl-proto (502), Dst Port: 47762 (47762), Seq: 84591917, Ack: 4166595040, Len: 0

It appears md5sum hashes are also incorrect.

Confirmed that the issues below are only related to erspan traffic. Tested with regular rspan and the issues do not occur.

• Incorrect MD5sum (file magic appears to be correct)
• Malformed packet field in eve log

OS information
Linux stplidsn02 4.2.3-1-ARCH #1 SMP PREEMPT Sat Oct 3 18:52:50 CEST 2015 x86_64 GNU/Linux
Suricata git ~ This is Suricata version 2.1dev (rev dcbbda5)

If developers need any further information or testing, please let me know.

Can you reproduce the issues with a pcap recording of your erspan traffic as well?

Jay MJ wrote:

Confirmed that the issues below are only related to erspan traffic. Tested with regular rspan and the issues do not occur.

Issues

• Incorrect MD5sum (file magic appears to be correct)
• Malformed packet field in eve log

OS information
Linux stplidsn02 4.2.3-1-ARCH #1 SMP PREEMPT Sat Oct 3 18:52:50 CEST 2015 x86_64 GNU/Linux
Suricata git ~ This is Suricata version 2.1dev (rev dcbbda5)

If developers need any further information or testing, please let me know.

Apologies, please disregard the hashing issue, that is a separate issue, and appears to be present across spans. I had made some configuration changes that probably borked that.

Victor Julien wrote:

Can you reproduce the issues with a pcap recording of your erspan traffic as well?

The pcaps suricata saves and tcpdump as well are the same, and OK. It appears to be how suricata parses the pcap.

What capture method are you using? If you're not using regular libpcap (-i on commandline) can you try that? Wondering if possibly there is something about our afpacket/pfring support that is related to this.

Victor Julien wrote:

What capture method are you using? If you're not using regular libpcap (-i on commandline) can you try that? Wondering if possibly there is something about our afpacket/pfring support that is related to this.

I was using af-packet, tried doing libpcap, same results happen.

I think it has to do with the erspan component of suricata. Plain rspan itself works fine, but once I revert to erspan, the packet alert field becomes malformed. What I think is happening is the packet alert field parsing is not addressing the proceeding 50 byte erspan header. Then, subsequently generated pcap gets improperly offset and encoded into the alert packet field of suricata.

Victor Julien wrote:

What capture method are you using? If you're not using regular libpcap (-i on commandline) can you try that? Wondering if possibly there is something about our afpacket/pfring support that is related to this.

Also tried pfring, the results are the same.

Looking at this some more, appears perhaps the GRE header itself could be a factor when packet field gets encoded for eve log? I'll do some more testing in time.

It looks like a couple of things are happening comparing to rspan and erspan and pf_ring on both. The alert packet field is encoding in base64 but I believe it does so by looking at the original encoded GRE/ERSPAN packet, so everything is malformed.

I also noticed that the bpf filter often doesn't work with this encoded traffic most of the time. In fact suricata won't pick up any traffic (unless bpf is something simple like tcp protocol). If you would like another ticket for this let me know.

• Mapping the packet field to properly decoded GRE/ESPAN packet rather than raw
• Apply bpf filters to properly decoded GRE/ERSPAN packet; not sure what it's using - perhaps also raw encoded original?

Not sure if there is much demand for erspan support, seems like I'm the only one using it. I also looked into rcdcap to handle decoding pre suricata with a tunnel interface, however the software is older, and I have a pending issue with the developer of that project.

Not sure if someone started working on this (I see some activity in git), but it appears that the md5sum is working now and correct (but sha1/256 are still incorrect). I am testing 3.2beta1, one erspan comes in stripped of erspan header and then passed to suricata, other erspan is fed straight to suricata. Strangely both md5 hashes match and are correct, however sha1 and sha256 have both different values and are not the correct hash of the file. This may be another bug, as my test shows even without erspan header the sha1/256 hashes are incorrect.

Also, the alert packet field decoded at initial look appears to be correct now, although I would like to do more research for a couple of days before calling that good.

If any additional testing needs to be done, don't hesitate to let me know.

The hash issue was fixed in 3.2.1, Bug #2004.

Still seeing malformed packet fields though (only with erspan traffic).

Jay MJ wrote:

The hash issue was fixed in 3.2.1, Bug #2004.

Still seeing malformed packet fields though (only with erspan traffic).

Still seeing the issue with malformed pcaps, packet / payload fields in eve logs with suricata 4.
Testing with rcdcap (virtual tap) which strips the erspan headers and everything works fine. Happy to do any testing / provide more info if needed.

Jay MJ wrote:

The hash issue was fixed in 3.2.1, Bug #2004.

Still seeing malformed packet fields though (only with erspan traffic).

Updating the proper ticket this time!

I have noticed that the only issue remaining in v 4.1.4 is the packet field in the eve log, which doesn't seem to be valid. The encoded packet field does not work when imported into scapy, or decoded and converted to a pcap file. Everything else appears to work properly (encoded payload, printed payload, recorded pcaps) with or without the erspan header.

Are you able to provide an input pcap as well as a rule that shows this issue?