Project

General

Profile

Actions

Optimization #7423

open

eve/json: reduce default memory buffer size; remove double buffering

Added by Jason Ish 23 days ago.

Status:
New
Priority:
Normal
Assignee:
Target version:
Effort:
Difficulty:
Label:

Description

Earlier the issue of log records being 2 large and truncated was addressed with a simpler fix suitable for backporting to 7. However, it is desirable to have some better options here, from https://redmine.openinfosecfoundation.org/issues/7300#note-6

[Victor Julien]

For 8, we are thinking of:
1. get rid double buffering
2. add limit to jsonbuilder in "root" object open call
3. jsonbuilder skips adding fields that would make it exceed the limit
4. jsonbuilder root object would hold a bit of info: like first field name that tripped the limit
5. special root object close call will return warning/error for logging, and close the partial object correctly

Additional thoughts: current 10MiB is very large and should probably be reduced. Also should perhaps be configurable. Maybe as a jb_set_limit call or similar.


Related issues 1 (0 open1 closed)

Related to Suricata - Bug #7300: output: oversized records lead to invalid jsonClosedJason IshActions
Actions #1

Updated by Jason Ish 23 days ago

  • Related to Bug #7300: output: oversized records lead to invalid json added
Actions

Also available in: Atom PDF