https://redmine.openinfosecfoundation.org/https://redmine.openinfosecfoundation.org/favicon.ico?17011170022018-12-04T23:43:12ZOpen Information Security FoundationSuricata - Bug #2726: writing large number of json events on high speed traffic results in packet dropshttps://redmine.openinfosecfoundation.org/issues/2726?journal_id=105952018-12-04T23:43:12ZPeter Manevpetermanev@gmail.com
<ul><li><strong>Subject</strong> changed from <i>writng large number of json events on high speed traffic results in packet drops</i> to <i>writing large number of json events on high speed traffic results in packet drops</i></li></ul> Suricata - Bug #2726: writing large number of json events on high speed traffic results in packet dropshttps://redmine.openinfosecfoundation.org/issues/2726?journal_id=110052019-02-18T10:39:43ZVictor Julienvictor@inliniac.net
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Assigned</i></li><li><strong>Assignee</strong> set to <i>Jason Ish</i></li><li><strong>Target version</strong> set to <i>70</i></li></ul><p>Overall, the generating and writing of so many events is simply and expensive operation. We can try to optimize things of course. I'm testing a branch for reducing heap allocations in logging certain fields.</p>
<p>Assigning ticket to Jason as he's looking into speeding up creating the JSON records.</p> Suricata - Bug #2726: writing large number of json events on high speed traffic results in packet dropshttps://redmine.openinfosecfoundation.org/issues/2726?journal_id=122912019-05-28T20:45:00ZAndreas Herzoisf@herzandreas.de
<ul></ul><p>As I also have some setups now where more and more logging result in higher drop rates is there any information that could help?</p>
<p><a class="user active user-mention" href="https://redmine.openinfosecfoundation.org/users/4993">@Peter Pan</a> did you try using sockets or even redis and compare the performance?</p> Suricata - Bug #2726: writing large number of json events on high speed traffic results in packet dropshttps://redmine.openinfosecfoundation.org/issues/2726?journal_id=123012019-05-28T21:27:22ZPeter Manevpetermanev@gmail.com
<ul></ul><p>There are other means - via socket/redis etc - the main point here is that (as Victor mentions above) - it is an expensive operation and there is a lot of contention (imagine 40 threads writing in the same file lots of events). There were a few ideas discussed earlier optimization/logfile per thread/ all writing being handled by diff set of threads etc I am guessing optimization is the first step. From what I see there is no such penalty when using Redis for example.</p> Suricata - Bug #2726: writing large number of json events on high speed traffic results in packet dropshttps://redmine.openinfosecfoundation.org/issues/2726?journal_id=139092019-09-26T10:00:02ZVictor Julienvictor@inliniac.net
<ul><li><strong>Target version</strong> changed from <i>70</i> to <i>6.0.0beta1</i></li></ul> Suricata - Bug #2726: writing large number of json events on high speed traffic results in packet dropshttps://redmine.openinfosecfoundation.org/issues/2726?journal_id=145792019-11-06T20:55:26ZVictor Julienvictor@inliniac.net
<ul></ul><p>This was shared in a Zeek ticket: <a class="external" href="https://github.com/miloyip/nativejson-benchmark#stringify-time">https://github.com/miloyip/nativejson-benchmark#stringify-time</a><br />Some Rust numbers <a class="external" href="https://github.com/serde-rs/json-benchmark">https://github.com/serde-rs/json-benchmark</a></p> Suricata - Bug #2726: writing large number of json events on high speed traffic results in packet dropshttps://redmine.openinfosecfoundation.org/issues/2726?journal_id=145802019-11-06T20:55:48ZVictor Julienvictor@inliniac.net
<ul><li><strong>Related to</strong> <i><a class="issue tracker-2 status-5 priority-4 priority-default closed" href="/issues/3293">Feature #3293</a>: eve: per thread output files</i> added</li></ul> Suricata - Bug #2726: writing large number of json events on high speed traffic results in packet dropshttps://redmine.openinfosecfoundation.org/issues/2726?journal_id=155132020-03-05T13:55:29ZJason Ishjason.ish@oisf.net
<ul><li><strong>Status</strong> changed from <i>Assigned</i> to <i>In Review</i></li></ul><p>Pull request: <a class="external" href="https://github.com/OISF/suricata/pull/4634">https://github.com/OISF/suricata/pull/4634</a></p> Suricata - Bug #2726: writing large number of json events on high speed traffic results in packet dropshttps://redmine.openinfosecfoundation.org/issues/2726?journal_id=168342020-07-06T18:57:54ZJason Ishjason.ish@oisf.net
<ul><li><strong>Status</strong> changed from <i>In Review</i> to <i>Closed</i></li></ul><p>Closing. JsonBuilder is now in master. We have a tracking ticket to track conversions of sub-systems from Jansson to JsonBuilder here:<br /><a class="external" href="https://redmine.openinfosecfoundation.org/issues/3707">https://redmine.openinfosecfoundation.org/issues/3707</a></p> Suricata - Bug #2726: writing large number of json events on high speed traffic results in packet dropshttps://redmine.openinfosecfoundation.org/issues/2726?journal_id=168352020-07-06T19:00:16ZVictor Julienvictor@inliniac.net
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-2 priority-4 priority-default parent" href="/issues/3707">Optimization #3707</a>: Convert JSON Loggers to JsonBuilder</i> added</li></ul> Suricata - Bug #2726: writing large number of json events on high speed traffic results in packet dropshttps://redmine.openinfosecfoundation.org/issues/2726?journal_id=203102021-07-07T12:23:29ZJuliana Fajardini Reichow
<ul><li><strong>Related to</strong> <i><a class="issue tracker-6 status-8 priority-4 priority-default" href="/issues/4557">Documentation #4557</a>: Add document about JsonBuilder</i> added</li></ul>