Project

General

Profile

Actions
Copy link

Feature #1208

closed

JSON Output Enhancement - Include Payload(s)

Added by Eoin Miller over 10 years ago. Updated almost 10 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Victor Julien
Target version:
2.1beta1
Effort:
Difficulty:
Label:

Description

Please update the JSON output so that packet payload and all other unified2 fields are capable of being output. Also, it would be super awesome if Suricata was able to directly connect to Elastic Search and POST the JSON directly in as new documents in the cluster. This would completely remove the need for unified2 and barnyard2 and allow people to use things like Kibana to go through all their alerting data.

Actions
Copy link
 #1

Updated by Victor Julien over 10 years ago

Eoin Miller wrote:

Also, it would be super awesome if Suricata was able to directly connect to Elastic Search and POST the JSON directly in as new documents in the cluster. This would completely remove the need for unified2 and barnyard2 and allow people to use things like Kibana to go through all their alerting data.

You mean not use Logstash? For getting stuff into ES we don't need u2+barnyard2.

Actions
Copy link
 #2

Updated by Victor Julien over 10 years ago

Wrt the ticket, some work is being done here https://github.com/inliniac/suricata/pull/922

Actions
Copy link
 #3

Updated by Eoin Miller over 10 years ago

Victor Julien wrote:

Eoin Miller wrote:

Also, it would be super awesome if Suricata was able to directly connect to Elastic Search and POST the JSON directly in as new documents in the cluster. This would completely remove the need for unified2 and barnyard2 and allow people to use things like Kibana to go through all their alerting data.

You mean not use Logstash? For getting stuff into ES we don't need u2+barnyard2.

You do need u2+barnyard because there isn't a way to just dump packet payload info out into the JSON (unless using patches below it would appear).

Actions
Copy link
 #4

Updated by Eoin Miller over 10 years ago

Also I think the current method of only writing output to the filesystem to the eve.json file requires you running ELK on the same host as the sensor as opposed to adding functionality for the output from Suricata to use the HTTP transport to push the JSON to the elasticsearch cluster:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-http.html

Actions
Copy link
 #5

Updated by Victor Julien over 10 years ago

Eoin Miller wrote:

Also I think the current method of only writing output to the filesystem to the eve.json file requires you running ELK on the same host as the sensor as opposed to adding functionality for the output from Suricata to use the HTTP transport to push the JSON to the elasticsearch cluster:

No, I use logstash-forwarder to ship the logs from my sensor to my ELK server. Easy to set up, encrypted comms and lightweight. Check https://github.com/elasticsearch/logstash-forwarder

Actions
Copy link
 #6

Updated by Victor Julien over 10 years ago

  • Subject changed from JSON Output Enhancement - Include Payload(s) + All Other unified2 Fields to JSON Output Enhancement - Include Payload(s)
  • Status changed from New to Closed
  • Assignee set to Victor Julien
  • Target version set to 3.0RC2
  • % Done changed from 0 to 100

Payloads implemented in https://github.com/inliniac/suricata/pull/1048

      types:
        - alert:
            payload: yes           # enable dumping payload in Base64
            # payload-printable: yes # enable dumping payload in printable (lossy) format
            # packet: yes            # enable dumping of packet (without stream segments)

For the 'all other unified2 fields' you need, please add (a) separate ticket(s).

Actions
Copy link
 #7

Updated by Victor Julien over 10 years ago

  • Target version changed from 3.0RC2 to 2.1beta1
Actions
Copy link
 #8

Updated by god lol almost 10 years ago

Is this feature available as part of some beta build/package or I should build it myself to test it?

Actions
Copy link
 #9

Updated by Peter Manev almost 10 years ago

That is available in 2.1beta2 (and 2.1beta1) - JSON Output Enhancemen Include Payload(s)

Actions
Copy link
 #10

Updated by god lol almost 10 years ago

Victor, would you mind sharing your logstash and forwarder configuration you're using with suricata?
I've got problem with json being escaped and not properly parsed when I'm trying to use such setup with suricata.

Actions
Copy link
 #12

Updated by god lol almost 10 years ago

Thank you, but I've already looked into wiki link - it refers to case where eve.json is read by logstash directly. My use-case involve logstash-forwarder and lumberjack protocol which is different.

Actions
Copy link

Also available in: Atom PDF