Optimization #4126
closedThreaded eve logging for output types other than regular file (socket, plugins, redis etc)
Added by Jason Ish almost 4 years ago. Updated over 3 years ago.
Updated by Jason Ish almost 4 years ago
- Related to Task #4097: Suricon 2020 brainstorm added
Updated by Jason Ish almost 4 years ago
- Assignee set to Jeff Lucovsky
- Target version set to 7.0.0-beta1
Updated by Andreas Herz over 3 years ago
- Subject changed from Threaded eve logging for output types other than regular file (socket, plugins, etc) to Threaded eve logging for output types other than regular file (socket, plugins, redis etc)
Updated by Andreas Herz over 3 years ago
This might be helpful for high performance setups that use redis, since it would help to scale the bottleneck with the single redis pipeline
Updated by Jeff Lucovsky over 3 years ago
- Status changed from New to In Progress
Updated by Jeff Lucovsky over 3 years ago
- Status changed from In Progress to In Review
Updated by Jason Ish over 3 years ago
Here are some draft notes on the life cycle of a plugin with this PR that I'm hoping we can get some more feedback on.
Callbacks¶
A file type output plugin consist of the following callbacks:- Init
- Close (TODO: Rename to Deinit?)
- ThreadInit
- ThreadDeinit
- Write
Init¶
static int Init(ConfNode *conf, bool threaded, void **data)
The Init
function is the first callback called. The arguments include the ConfNode
for the Eve output using the plugin, a flag to tell if the Eve output is multi-threaded or not and an output pointer to provide any data to future callbacks.
As threading of the output is not under the control of the plugin, the plugin needs to decide if it can handle being threaded or not. If the plugin cannot handle being threaded it should return from Init
with an error code -1
.
Or if the plugin cannot run unthreaded, it could return an error code if not threaded.
Ideally output plugins can handle both cases.
In non-threaded mode this is where the plugin might open a file, a database connection, a socket, etc. The next call into the plugin will the be the Write
callback along with the data
returned by this plugin.
In threaded mode this is where the plugin might setup any context that needs to be shared among all output threads, such as data provided in the ConfNode
.
ThreadInit¶
static int ThreadInit(void *init_data, int thread_id, void **thread_data)
In threaded mode this function will be called for each output thread. The arguments include the data
as returned by Init
in the init_data
field, the thread_id
of this output, and an output pointer thread_data
where context data specific to this thread can be stored.
The thread_id
is sequential and starts at 1, however there might be gaps, ie) 1..13,15,17.
When in threaded mode this is where the plugin would likely open a file, database connection, socket, etc.
Write¶
static int Write(const char *buffer, int len, void *data, void *thread_data)
The Write
callback is called for each event where buffer
is a formatted JSON string of the event (nul terminated string), len
is the length of this string, data
is the context data provided in Init
and thread_data
is the per-thread context data provided by ThreadInit
.
ThreadDeinit¶
static int ThreadDeinit(void *ctx, void *thread_data)
ThreadDeinit
will be called when an output thread is closed. This should free any resources allocated in ThreadInit
.
TODO: This should probably return void.
Close¶
static int Close(void *data)
Close
will be the final call into the plugin where it should cleanup any extra resources, such as those allocated in Init
and provided in the data
argument.
TODO: Rename to Deinit to match Init?
TODO: Probably should return void.
Updated by Danny Browning over 3 years ago
Would it be possible to always call ThreadInit/ThreadDeinit regardless of non-threaded or threaded mode? Plugin flow would always be Init -> ThreadInit -> Write (Repeat) -> ThreadDeinit -> Close.
Also ThreadInit should receive the previously created data from Init as a parameter.
Updated by Danny Browning over 3 years ago
Just realized ctx was the Init data. Maybe a rename to init_data or init_ctx?
Updated by Jeff Lucovsky over 3 years ago
Would it be possible to always call ThreadInit/ThreadDeinit regardless of non-threaded or threaded mode? Plugin flow would always be Init -> ThreadInit -> Write (Repeat) -> ThreadDeinit -> Close.
Yes -- this is the intended flow.
Also ThreadInit should receive the previously created data from Init as a parameter.
It does -- "The arguments include the data as returned by Init in the ctx field"
Just realized ctx was the Init data. Maybe a rename to init_data or init_ctx?
Yes --- this can be done.
Updated by Jason Ish over 3 years ago
Is threaded
still a useful argument to have in Init
? If the plugin is written in a thread safe manner (even just a lock around the same resource) then we don't need to worry about it too much. If the plugin really needs to know, it can call ConfGeBool(conf_node, "threaded")
.
Updated by Jeff Lucovsky over 3 years ago
Is threaded still a useful argument to have in Init? If the plugin is written in a thread safe manner (even just a lock around the same resource) then we don't need to worry about it too much. If the plugin really needs to know, it can call ConfGeBool(conf_node, "threaded").
I kept it in the interface to Init based on an earlier discussion we had.
You're correct -- it can decide the type of behavior to implement using ConfGetBool(..., "threaded") -- i'll update my pr
Updated by Jason Ish over 3 years ago
Yeah, I think it was more important when ThreadInit would not be called if unthreaded. But with ThreadInit being called no matter what, I think its less important.
Updated by Jeff Lucovsky over 3 years ago
Would it be possible to always call ThreadInit/ThreadDeinit regardless of non-threaded or threaded mode? Plugin flow would always be Init -> ThreadInit -> Write (Repeat) -> ThreadDeinit -> Close.
Yes -- this is the intended flow.
Hmmm ... I've polished things a bit and I don't think it'll be possible to do this since Suricata only needs calls ThreadInit/ThreadDeinit when the output logging is in threaded mode as there's additional context it stores.
Could you structure your Init function as
int Init(ConfNode *conf, bool threaded, void **init_data)) { . . if (!threaded) MyThreadInit() . . } int Deinit(void *init_data)) { if (!threaded) MyThreadDeinit() . . }
Updated by Jason Ish over 3 years ago
Suricata may only need to call ThreadInit when it is threaded mode, but does that mean it can't also call it while not threaded mode? I guess when not in threaded mode, the support bits for passing the thread_data around are not there?
Updated by Danny Browning over 3 years ago
Yeah, can definitely structure plugins that way, was just seeing if we could simplify the plugin writer experience.
Can write the example plugin that way if we can't easily always call thread init/deinit.
Updated by Jeff Lucovsky over 3 years ago
Suricata may only need to call ThreadInit when it is threaded mode, but does that mean it can't also call it while not threaded mode? I guess when not in threaded mode, the support bits for passing the thread_data around are not there?
Yes ... The code paths are quite different in Suricata with threaded mode.
Updated by Jeff Lucovsky over 3 years ago
- Status changed from In Review to Closed