Suricata

I agree with the Feature request -
although for the majority of outputs we are slowly but surely migrating towards JSON.
ex - eve.json

As far as configuration is concerned - it is yaml, do you mean some other config options?

The outputs - in the form of triggers of rules - are in my mind for the short term, good to keep in a variance (alot use unified2 with barnyard for example). I was thinking more about the case of Feature #249 as an example. This is a feature that with implement a new way of giving some input / configuration to suricata. These kinds of input situations should be kept uniform (eg, a concrete decision of for ex. yaml). And other forms of output that are none-rule-trigger/alert related should also be kept uniform with for ex json (packet profiling output, statistics, etc).

With flexibility in mind, and that many users today may have allready produces scripts and other solutions to convert the data in http, dns, stats and so on to a more workable structure, could there maybe be implemented a possibilty to choose format until the "old" is removed in a future release?

Example:

- http-log:
    enabled: yes/no
    filename: <insert filename>
    append: yes/no
    format: json/csv/"old_standard" 

1. A proposal of a shift of policy where all input / output formats are standardized with the intention of that the chosen format should be easy to "send on to" other applications (ex: json, cef, csv)
2. A proposal of a shift towards incorporation of JSON at broader level than today.

So what you are proposing is for the "old" style logging - to be able to chose your output format?
So that you can enable different file outputs for the different logs (dns,http,tls...), is that correct?

If that is the case - what stops you from using the eve.log with the JSON format?

Im not just talking about logging, i'm talking about all the choices made for formats that are read / outputed that not the alerts/rules/triggers (many systems use unified2 with barnyard and maybe the old logging, would be bad to take that away). Im takling about configuration, smaller inputs like Feature #249 and the other outputs that are not eve.log / fast-log / unified2. That some decision should be made to say "ok people, we are using YAML for all configurations, and all non-alert related output (eg DNS logging, HTTP logging) should be JSON" as an example.

Yes it is possible to get DNS and HTTP logging in the Eve-log, but with systems that have huge amounts of traffic it is not very doable to have all of this data going to one place. An example of a configuration i'm using is HTTP traffic goes to http.log files and are rotated (or else the files get huge) and parsed every 5 min. Alerts are writen in the unified2 format and further processed by barnyard. The stats.log is rotated every hour and parsed every 30 seconds and can then be passed on to graphite or an ELK stack. DNS logging is rotated just as much as HTTP logging but parsed and sent to different systems. So many secondary systems (reparsing due to the format of the data as it is produced from suricata) have to be implemented for get all this to work. Even though a system say can handle direct input of JSON data, it is much easier to process such a common standard than some self-created-format.

The JSON output can easily be directed to seperate files. I did a quick write up here: EveJSONOutput.

Even though the JSON format is generally quite self explaining, we've started documenting the fields here: EveJSONFormat. As you can see, it's far from done.

Ahh! Well that is great! Did not know about that ability. I updated the EveJSONFormat to describe the even_types for HTTP and DNS.