Project

General

Profile

Actions

Feature #1208

closed

JSON Output Enhancement - Include Payload(s)

Added by Eoin Miller over 8 years ago. Updated almost 8 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Effort:
Difficulty:
Label:

Description

Please update the JSON output so that packet payload and all other unified2 fields are capable of being output. Also, it would be super awesome if Suricata was able to directly connect to Elastic Search and POST the JSON directly in as new documents in the cluster. This would completely remove the need for unified2 and barnyard2 and allow people to use things like Kibana to go through all their alerting data.

Actions #1

Updated by Victor Julien over 8 years ago

Eoin Miller wrote:

Also, it would be super awesome if Suricata was able to directly connect to Elastic Search and POST the JSON directly in as new documents in the cluster. This would completely remove the need for unified2 and barnyard2 and allow people to use things like Kibana to go through all their alerting data.

You mean not use Logstash? For getting stuff into ES we don't need u2+barnyard2.

Actions #2

Updated by Victor Julien over 8 years ago

Wrt the ticket, some work is being done here https://github.com/inliniac/suricata/pull/922

Actions #3

Updated by Eoin Miller over 8 years ago

Victor Julien wrote:

Eoin Miller wrote:

Also, it would be super awesome if Suricata was able to directly connect to Elastic Search and POST the JSON directly in as new documents in the cluster. This would completely remove the need for unified2 and barnyard2 and allow people to use things like Kibana to go through all their alerting data.

You mean not use Logstash? For getting stuff into ES we don't need u2+barnyard2.

You do need u2+barnyard because there isn't a way to just dump packet payload info out into the JSON (unless using patches below it would appear).

Actions #4

Updated by Eoin Miller over 8 years ago

Also I think the current method of only writing output to the filesystem to the eve.json file requires you running ELK on the same host as the sensor as opposed to adding functionality for the output from Suricata to use the HTTP transport to push the JSON to the elasticsearch cluster:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-http.html

Actions #5

Updated by Victor Julien over 8 years ago

Eoin Miller wrote:

Also I think the current method of only writing output to the filesystem to the eve.json file requires you running ELK on the same host as the sensor as opposed to adding functionality for the output from Suricata to use the HTTP transport to push the JSON to the elasticsearch cluster:

No, I use logstash-forwarder to ship the logs from my sensor to my ELK server. Easy to set up, encrypted comms and lightweight. Check https://github.com/elasticsearch/logstash-forwarder

Actions #6

Updated by Victor Julien about 8 years ago

  • Subject changed from JSON Output Enhancement - Include Payload(s) + All Other unified2 Fields to JSON Output Enhancement - Include Payload(s)
  • Status changed from New to Closed
  • Assignee set to Victor Julien
  • Target version set to 3.0RC2
  • % Done changed from 0 to 100

Payloads implemented in https://github.com/inliniac/suricata/pull/1048

      types:
        - alert:
            payload: yes           # enable dumping payload in Base64
            # payload-printable: yes # enable dumping payload in printable (lossy) format
            # packet: yes            # enable dumping of packet (without stream segments)

For the 'all other unified2 fields' you need, please add (a) separate ticket(s).

Actions #7

Updated by Victor Julien about 8 years ago

  • Target version changed from 3.0RC2 to 2.1beta1
Actions #8

Updated by god lol almost 8 years ago

Is this feature available as part of some beta build/package or I should build it myself to test it?

Actions #9

Updated by Peter Manev almost 8 years ago

That is available in 2.1beta2 (and 2.1beta1) - JSON Output Enhancemen Include Payload(s)

Actions #10

Updated by god lol almost 8 years ago

Victor, would you mind sharing your logstash and forwarder configuration you're using with suricata?
I've got problem with json being escaped and not properly parsed when I'm trying to use such setup with suricata.

Actions #12

Updated by god lol almost 8 years ago

Thank you, but I've already looked into wiki link - it refers to case where eve.json is read by logstash directly. My use-case involve logstash-forwarder and lumberjack protocol which is different.

Actions

Also available in: Atom PDF