Project

General

Profile

Actions

Bug #1444

closed

EVE output writing duplicate and malformed events.

Added by Brandon Lattin almost 9 years ago. Updated almost 9 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
Affected Versions:
Effort:
Difficulty:
Label:

Description

EVE json files occasionally write duplicates and/or merge events together in a malformed blob.

I've included to snippets from an EVE output. The samples are not redacted. Please do not distribute beyond oisf (and not on the mailing list please).

Search (eve.json) for the following to see the event merger:
WtHWkpLVFF{"timestamp

eve2.json has an additional example.

System specs:

RHEL 6.6
Linux xxxx 2.6.32-504.12.2.el6.x86_64 #1 SMP Sun Feb 1 12:14:02 EST 2015 x86_64 x86_64 x86_64 GNU/Linux

jansson.x86_64 2.6-1.el6 @epel
jansson-devel.x86_64 2.6-1.el6 @epel
jansson.i686 2.6-1.el6 epel
jansson-devel.i686 2.6-1.el6 epel

suricata-2.1beta2

suricata.yaml EVE config:

- eve-log:
append: yes
enabled: yes
type: file #file|syslog|unix_dgram|unix_stream
filename: eve-port0.json # the following are valid when type: syslog above
#identity: "suricata"
#facility: local5
#level: Info ## possible levels: Emergency, Alert, Critical, ## Error, Warning, Notice, Info, Debug
types:
- alert:
payload: yes # enable dumping payload in Base64
payload-printable: yes # enable dumping payload in printable (lossy) format
packet: yes # enable dumping of packet (without stream segments)
http: no # enable dumping of http fields

I've since turned off both the "payload" and "packet" options and have yet to see another event merger, but it's only been a couple of hours since the change.


Files

eve.json (258 KB) eve.json EVE output duplication and malformed events Brandon Lattin, 04/08/2015 01:14 PM
eve2.json (155 KB) eve2.json Brandon Lattin, 04/08/2015 01:38 PM
Actions #1

Updated by Victor Julien almost 9 years ago

Can you try beta3 or the git master. We've made some changes since beta2.

Actions #2

Updated by Brandon Lattin almost 9 years ago

Victor Julien wrote:

Can you try beta3 or the git master. We've made some changes since beta2.

I can. I'll upgrade our sensors later this afternoon and give an update tomorrow.

Actions #3

Updated by Brandon Lattin almost 9 years ago

Running Suricata-2.1beta3 seems to have cleared up the event merger problem (only about a 1h uptime; I'll update if I see the problem again), but I'm still seeing some strange behavior.

1. Events that trigger off the same packet formats alert 1 properly and then alerts n+1.. with the "alert" KV pair out of order. See the paste below for an example.

{"timestamp":"2015-04-10T10:48:42.309235","flow_id":108311248,"in_iface":"snf0","event_type":"alert","vlan":3740,"src_ip":"<redacted>","src_port":2892,"dest_ip":"<redacted>","dest_port":80,"proto":"TCP","alert":{"action":"allowed","gid":1,"signature_id":2016977,"rev":3,"signature":"ET WEB_SERVER allow_url_include PHP config option in uri","category":"A Network Trojan was Detected","severity":1,"tx_id":0},"payload":"<redacted>"","payload_printable":"<redacted>"","stream":1,"packet":"<redacted>""}
{"timestamp":"2015-04-10T10:48:42.309235","flow_id":108311248,"in_iface":"snf0","event_type":"alert","vlan":3740,"src_ip":"<redacted>","src_port":2892,"dest_ip":"<redacted>","dest_port":80,"proto":"TCP","payload":"<redacted>"","payload_printable":"<redacted>"","stream":1,"packet":"<redacted>"","alert":{"action":"allowed","gid":1,"signature_id":2016978,"rev":3,"signature":"ET WEB_SERVER safe_mode PHP config option in uri","category":"A Network Trojan was Detected","severity":1,"tx_id":0}}
{"timestamp":"2015-04-10T10:48:42.309235","flow_id":108311248,"in_iface":"snf0","event_type":"alert","vlan":3740,"src_ip":"<redacted>","src_port":2892,"dest_ip":"<redacted>","dest_port":80,"proto":"TCP","payload":"<redacted>"","payload_printable":"<redacted>"","stream":1,"packet":"<redacted>"","alert":{"action":"allowed","gid":1,"signature_id":2016979,"rev":4,"signature":"ET WEB_SERVER suhosin.simulation PHP config option in uri","category":"A Network Trojan was Detected","severity":1,"tx_id":0}}
{"timestamp":"2015-04-10T10:48:42.309235","flow_id":108311248,"in_iface":"snf0","event_type":"alert","vlan":3740,"src_ip":"<redacted>","src_port":2892,"dest_ip":"<redacted>","dest_port":80,"proto":"TCP","payload":"<redacted>"","payload_printable":"<redacted>"","stream":1,"packet":"<redacted>"","alert":{"action":"allowed","gid":1,"signature_id":2016980,"rev":5,"signature":"ET WEB_SERVER disable_functions PHP config option in uri","category":"A Network Trojan was Detected","severity":1,"tx_id":0}}
{"timestamp":"2015-04-10T10:48:42.309235","flow_id":108311248,"in_iface":"snf0","event_type":"alert","vlan":3740,"src_ip":"<redacted>","src_port":2892,"dest_ip":"<redacted>","dest_port":80,"proto":"TCP","payload":"<redacted>"","payload_printable":"<redacted>"","stream":1,"packet":"<redacted>"","alert":{"action":"allowed","gid":1,"signature_id":2016981,"rev":4,"signature":"ET WEB_SERVER open_basedir PHP config option in uri","category":"A Network Trojan was Detected","severity":1,"tx_id":0}}
{"timestamp":"2015-04-10T10:48:42.309235","flow_id":108311248,"in_iface":"snf0","event_type":"alert","vlan":3740,"src_ip":"<redacted>","src_port":2892,"dest_ip":"<redacted>","dest_port":80,"proto":"TCP","payload":"<redacted>"","payload_printable":"<redacted>"","stream":1,"packet":"<redacted>"","alert":{"action":"allowed","gid":1,"signature_id":2016982,"rev":3,"signature":"ET WEB_SERVER auto_prepend_file PHP config option in uri","category":"A Network Trojan was Detected","severity":1,"tx_id":0}}

2. On a whim, I set "payload: no" and "packet: no" keeping only "payload-printable: yes" enabled. While this did not clear up the out-of-order "alert" KV pair, it did have some other unexpected behavior. The "vlan" and "in_iface" KV pairs disappeared. I'm not sure if this is expected behavior when setting "packet: no", but it seems odd to lose that information. See the paste below for details.

{"timestamp":"2015-04-09T16:00:15.780114","flow_id":125911712,"event_type":"alert","src_ip":"<redacted>","src_port":57572,"dest_ip":"<redacted>","dest_port":80,"proto":"TCP","alert":{"action":"allowed","gid":1,"signature_id":2011768,"rev":6,"signature":"ET WEB_SERVER PHP tags in HTTP POST","category":"Web Application Attack","severity":1,"tx_id":0},"payload_printable":"<redacted>"","stream":1}
{"timestamp":"2015-04-09T16:00:15.780114","flow_id":125911712,"event_type":"alert","src_ip":"<redacted>","src_port":57572,"dest_ip":"<redacted>","dest_port":80,"proto":"TCP","payload_printable":"<redacted>"","stream":1,"alert":{"action":"allowed","gid":1,"signature_id":2016977,"rev":3,"signature":"ET WEB_SERVER allow_url_include PHP config option in uri","category":"A Network Trojan was Detected","severity":1,"tx_id":0}}
{"timestamp":"2015-04-09T16:00:15.780114","flow_id":125911712,"event_type":"alert","src_ip":"<redacted>","src_port":57572,"dest_ip":"<redacted>","dest_port":80,"proto":"TCP","payload_printable":"<redacted>"","stream":1,"alert":{"action":"allowed","gid":1,"signature_id":2016978,"rev":3,"signature":"ET WEB_SERVER safe_mode PHP config option in uri","category":"A Network Trojan was Detected","severity":1,"tx_id":0}}
{"timestamp":"2015-04-09T16:00:15.780114","flow_id":125911712,"event_type":"alert","src_ip":"<redacted>","src_port":57572,"dest_ip":"<redacted>","dest_port":80,"proto":"TCP","payload_printable":"<redacted>"","stream":1,"alert":{"action":"allowed","gid":1,"signature_id":2016979,"rev":4,"signature":"ET WEB_SERVER suhosin.simulation PHP config option in uri","category":"A Network Trojan was Detected","severity":1,"tx_id":0}}
{"timestamp":"2015-04-09T16:00:15.780114","flow_id":125911712,"event_type":"alert","src_ip":"<redacted>","src_port":57572,"dest_ip":"<redacted>","dest_port":80,"proto":"TCP","payload_printable":"<redacted>"","stream":1,"alert":{"action":"allowed","gid":1,"signature_id":2016980,"rev":5,"signature":"ET WEB_SERVER disable_functions PHP config option in uri","category":"A Network Trojan was Detected","severity":1,"tx_id":0}}
{"timestamp":"2015-04-09T16:00:15.780114","flow_id":125911712,"event_type":"alert","src_ip":"<redacted>","src_port":57572,"dest_ip":"<redacted>","dest_port":80,"proto":"TCP","payload_printable":"<redacted>"","stream":1,"alert":{"action":"allowed","gid":1,"signature_id":2016981,"rev":4,"signature":"ET WEB_SERVER open_basedir PHP config option in uri","category":"A Network Trojan was Detected","severity":1,"tx_id":0}}
{"timestamp":"2015-04-09T16:00:15.780114","flow_id":125911712,"event_type":"alert","src_ip":"<redacted>","src_port":57572,"dest_ip":"<redacted>","dest_port":80,"proto":"TCP","payload_printable":"<redacted>"","stream":1,"alert":{"action":"allowed","gid":1,"signature_id":2016982,"rev":3,"signature":"ET WEB_SERVER auto_prepend_file PHP config option in uri","category":"A Network Trojan was Detected","severity":1,"tx_id":0}}
Actions #4

Updated by Victor Julien almost 9 years ago

  • Status changed from New to Closed

I don't think json keeps an order in the k/v's, so whatever libjansson decides for us there is fine. Won't make a difference for the automated processing tools anyway.

Actions

Also available in: Atom PDF