https://redmine.openinfosecfoundation.org/https://redmine.openinfosecfoundation.org/favicon.ico?17011170022014-06-29T13:37:27ZOpen Information Security FoundationSuricata - Feature #1228: Suricata stats.log in JSON formathttps://redmine.openinfosecfoundation.org/issues/1228?journal_id=44222014-06-29T13:37:27ZPeter Manevpetermanev@gmail.com
<ul></ul><p>Similar to <br /><a class="external" href="https://redmine.openinfosecfoundation.org/issues/1036">https://redmine.openinfosecfoundation.org/issues/1036</a></p> Suricata - Feature #1228: Suricata stats.log in JSON formathttps://redmine.openinfosecfoundation.org/issues/1228?journal_id=47072014-10-21T15:56:19ZAndreas Moemoe.andreas@gmail.com
<ul></ul><p>Started working on a branch for this. Just a simple stats.format key in the yaml file (as shown above), and using the current data available in the counters.c file. Using libjansson for the JSON creation. Any suggestions to JSON-Structure? I was thinking something like:</p>
<pre>
{
"metadata": {
"date": "21/10/2014",
"uptime": {
"days": 1,
"hours": 2,
"minutes": 3,
"seconds": 4,
},
},
"Counters": {
"TM Name": {
"counter": Value,
"counter": Value,
"counter": Value,
"counter": Value
},
"TM Name": {
"counter": Value,
"counter": Value,
"counter": Value,
"counter": Value
}
}
}
</pre> Suricata - Feature #1228: Suricata stats.log in JSON formathttps://redmine.openinfosecfoundation.org/issues/1228?journal_id=47092014-10-22T04:03:01ZVictor Julienvictor@inliniac.net
<ul><li><strong>Target version</strong> set to <i>3.0RC2</i></li></ul><p>Great that you want to have a look Andreas. However, quite a bit of code already exists here: <a class="external" href="https://github.com/inliniac/suricata/pull/1010">https://github.com/inliniac/suricata/pull/1010</a></p> Suricata - Feature #1228: Suricata stats.log in JSON formathttps://redmine.openinfosecfoundation.org/issues/1228?journal_id=47102014-10-22T05:10:13ZAndreas Moemoe.andreas@gmail.com
<ul></ul><p>Ahh, didnt see that pull request. Looks like alot has been done. Is the wish for a Stats logger API (as you Victor commented on the pull request) what is stopping it from beeing merged or are there any other missing features / bugs?</p> Suricata - Feature #1228: Suricata stats.log in JSON formathttps://redmine.openinfosecfoundation.org/issues/1228?journal_id=47112014-10-22T05:14:21ZVictor Julienvictor@inliniac.net
<ul></ul><p>The stats logger api doesn't exist yet, so this will have to be created first.</p> Suricata - Feature #1228: Suricata stats.log in JSON formathttps://redmine.openinfosecfoundation.org/issues/1228?journal_id=47122014-10-22T09:48:20ZTom DeCaniodecanio.tom@gmail.com
<ul></ul><p>Andreas. I'd be happy to work with you to finish this. I was hoping that Victor would propose a stats logger api. I ran with what was there. Perhaps Victor can propose something and Andreas can help me implement it on top of what has been done already.</p> Suricata - Feature #1228: Suricata stats.log in JSON formathttps://redmine.openinfosecfoundation.org/issues/1228?journal_id=47632014-11-05T06:40:24ZVictor Julienvictor@inliniac.net
<ul></ul><p>The stats API implemented in <a class="external" href="https://github.com/inliniac/suricata/pull/1200">https://github.com/inliniac/suricata/pull/1200</a> and Tom's JSON output based on that in <a class="external" href="https://github.com/inliniac/suricata/pull/1202">https://github.com/inliniac/suricata/pull/1202</a></p>
<p>I'm not happy with the structure though. I've tried both Kibana and Graphite (fed by Logstash), but neither can easily take it in. One of the issues is that the "TM Name" is different for each capture method, capture config and runmode. So having a dashboard that works for all is going to be impossible, while this is normally the strength of our eve output.</p>
<p>When feeding Logstash to Graphite I found that it didn't really like the nesting (e.g. tcp:{syn:10} for counter tcp.syn). But when using the 'fields_are_metrics' option it didn't like the wrapping of it all in 'stats' either.</p>
<p>Minor disappointment was that Kibana 3 couldn't display multiple data series in the historam view (e.g. decoder.pkts and decoder.invalid), although this is likely to be added in Kibana 4.</p>
<p>So I think what we need is either different format that 'just works', or we should supply decent example configs for making good use of this in Kibana and/or Graphite.</p> Suricata - Feature #1228: Suricata stats.log in JSON formathttps://redmine.openinfosecfoundation.org/issues/1228?journal_id=47892014-11-09T12:16:16ZAndreas Moemoe.andreas@gmail.com
<ul></ul><p>Feature <a class="issue tracker-2 status-5 priority-3 priority-lowest closed" title="Feature: AF_Packet summary stats in stats.log (Closed)" href="https://redmine.openinfosecfoundation.org/issues/490">#490</a> looks at the issue of non-aggregated stats. Could (with the implementation you have given Victor) this be included now, by say a flag in the YAML (eg. aggregated yes/no) and sum up the similar counter values?</p> Suricata - Feature #1228: Suricata stats.log in JSON formathttps://redmine.openinfosecfoundation.org/issues/1228?journal_id=52802015-05-27T10:24:51ZVictor Julienvictor@inliniac.net
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Closed</i></li><li><strong>Assignee</strong> set to <i>Victor Julien</i></li><li><strong>Target version</strong> changed from <i>3.0RC2</i> to <i>3.0RC1</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li></ul><p><a class="external" href="https://github.com/inliniac/suricata/pull/1508">https://github.com/inliniac/suricata/pull/1508</a></p>