Feature #1208
closedJSON Output Enhancement - Include Payload(s)
Added by Eoin Miller almost 10 years ago. Updated over 9 years ago.
Description
Please update the JSON output so that packet payload and all other unified2 fields are capable of being output. Also, it would be super awesome if Suricata was able to directly connect to Elastic Search and POST the JSON directly in as new documents in the cluster. This would completely remove the need for unified2 and barnyard2 and allow people to use things like Kibana to go through all their alerting data.
Updated by Victor Julien almost 10 years ago
Eoin Miller wrote:
Also, it would be super awesome if Suricata was able to directly connect to Elastic Search and POST the JSON directly in as new documents in the cluster. This would completely remove the need for unified2 and barnyard2 and allow people to use things like Kibana to go through all their alerting data.
You mean not use Logstash? For getting stuff into ES we don't need u2+barnyard2.
Updated by Victor Julien almost 10 years ago
Wrt the ticket, some work is being done here https://github.com/inliniac/suricata/pull/922
Updated by Eoin Miller almost 10 years ago
Victor Julien wrote:
Eoin Miller wrote:
Also, it would be super awesome if Suricata was able to directly connect to Elastic Search and POST the JSON directly in as new documents in the cluster. This would completely remove the need for unified2 and barnyard2 and allow people to use things like Kibana to go through all their alerting data.
You mean not use Logstash? For getting stuff into ES we don't need u2+barnyard2.
You do need u2+barnyard because there isn't a way to just dump packet payload info out into the JSON (unless using patches below it would appear).
Updated by Eoin Miller almost 10 years ago
Also I think the current method of only writing output to the filesystem to the eve.json file requires you running ELK on the same host as the sensor as opposed to adding functionality for the output from Suricata to use the HTTP transport to push the JSON to the elasticsearch cluster:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-http.html
Updated by Victor Julien almost 10 years ago
Eoin Miller wrote:
Also I think the current method of only writing output to the filesystem to the eve.json file requires you running ELK on the same host as the sensor as opposed to adding functionality for the output from Suricata to use the HTTP transport to push the JSON to the elasticsearch cluster:
No, I use logstash-forwarder to ship the logs from my sensor to my ELK server. Easy to set up, encrypted comms and lightweight. Check https://github.com/elasticsearch/logstash-forwarder
Updated by Victor Julien over 9 years ago
- Subject changed from JSON Output Enhancement - Include Payload(s) + All Other unified2 Fields to JSON Output Enhancement - Include Payload(s)
- Status changed from New to Closed
- Assignee set to Victor Julien
- Target version set to 3.0RC2
- % Done changed from 0 to 100
Payloads implemented in https://github.com/inliniac/suricata/pull/1048
types:
- alert:
payload: yes # enable dumping payload in Base64
# payload-printable: yes # enable dumping payload in printable (lossy) format
# packet: yes # enable dumping of packet (without stream segments)
For the 'all other unified2 fields' you need, please add (a) separate ticket(s).
Updated by Victor Julien over 9 years ago
- Target version changed from 3.0RC2 to 2.1beta1
Updated by god lol over 9 years ago
Is this feature available as part of some beta build/package or I should build it myself to test it?
Updated by Peter Manev over 9 years ago
That is available in 2.1beta2 (and 2.1beta1) - JSON Output Enhancemen Include Payload(s)
Updated by god lol over 9 years ago
Victor, would you mind sharing your logstash and forwarder configuration you're using with suricata?
I've got problem with json being escaped and not properly parsed when I'm trying to use such setup with suricata.
Updated by Peter Manev over 9 years ago
There is one in our wiki that you can find here:
https://redmine.openinfosecfoundation.org/projects/suricata/wiki/_Logstash_Kibana_and_Suricata_JSON_output
hope it helps
Updated by god lol over 9 years ago
Thank you, but I've already looked into wiki link - it refers to case where eve.json is read by logstash directly. My use-case involve logstash-forwarder and lumberjack protocol which is different.