Project

General

Profile

Actions

Optimization #7423

open
JI OD

eve/json: reduce default memory buffer size; remove double buffering

Optimization #7423: eve/json: reduce default memory buffer size; remove double buffering

Added by Jason Ish over 1 year ago. Updated 3 months ago.

Status:
Assigned
Priority:
Normal
Assignee:
Target version:
Effort:
Difficulty:
Label:

Description

Earlier the issue of log records being 2 large and truncated was addressed with a simpler fix suitable for backporting to 7. However, it is desirable to have some better options here, from https://redmine.openinfosecfoundation.org/issues/7300#note-6

[Victor Julien]

For 8, we are thinking of:
1. get rid double buffering
2. add limit to jsonbuilder in "root" object open call
3. jsonbuilder skips adding fields that would make it exceed the limit
4. jsonbuilder root object would hold a bit of info: like first field name that tripped the limit
5. special root object close call will return warning/error for logging, and close the partial object correctly

Additional thoughts: current 10MiB is very large and should probably be reduced. Also should perhaps be configurable. Maybe as a jb_set_limit call or similar.


Related issues 1 (0 open1 closed)

Related to Suricata - Bug #7300: output: oversized records lead to invalid jsonClosedJason IshActions

JI Updated by Jason Ish over 1 year ago Actions #1

  • Related to Bug #7300: output: oversized records lead to invalid json added

VJ Updated by Victor Julien over 1 year ago Actions #2

  • Target version changed from 8.0.0-beta1 to 9.0.0-beta1

PA Updated by Philippe Antoine 3 months ago Actions #3

  • Status changed from New to Assigned
Actions

Also available in: PDF Atom