Project

General

Profile

Actions

Optimization #4126

closed

Threaded eve logging for output types other than regular file (socket, plugins, redis etc)

Added by Jason Ish about 4 years ago. Updated over 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Effort:
Difficulty:
Label:

Related issues 1 (1 open0 closed)

Related to Suricata - Task #4097: Suricon 2020 brainstormAssignedVictor JulienActions
Actions #1

Updated by Jason Ish about 4 years ago

  • Related to Task #4097: Suricon 2020 brainstorm added
Actions #2

Updated by Jason Ish about 4 years ago

  • Assignee set to Jeff Lucovsky
  • Target version set to 7.0.0-beta1
Actions #3

Updated by Andreas Herz almost 4 years ago

  • Subject changed from Threaded eve logging for output types other than regular file (socket, plugins, etc) to Threaded eve logging for output types other than regular file (socket, plugins, redis etc)
Actions #4

Updated by Andreas Herz almost 4 years ago

This might be helpful for high performance setups that use redis, since it would help to scale the bottleneck with the single redis pipeline

Actions #5

Updated by Jeff Lucovsky almost 4 years ago

  • Status changed from New to In Progress
Actions #6

Updated by Jeff Lucovsky almost 4 years ago

  • Status changed from In Progress to In Review
Actions #7

Updated by Jason Ish over 3 years ago

Here are some draft notes on the life cycle of a plugin with this PR that I'm hoping we can get some more feedback on.

Callbacks

A file type output plugin consist of the following callbacks:
  • Init
  • Close (TODO: Rename to Deinit?)
  • ThreadInit
  • ThreadDeinit
  • Write

Init

static int Init(ConfNode *conf, bool threaded, void **data)

The Init function is the first callback called. The arguments include the ConfNode for the Eve output using the plugin, a flag to tell if the Eve output is multi-threaded or not and an output pointer to provide any data to future callbacks.

As threading of the output is not under the control of the plugin, the plugin needs to decide if it can handle being threaded or not. If the plugin cannot handle being threaded it should return from Init with an error code -1.

Or if the plugin cannot run unthreaded, it could return an error code if not threaded.

Ideally output plugins can handle both cases.

In non-threaded mode this is where the plugin might open a file, a database connection, a socket, etc. The next call into the plugin will the be the Write callback along with the data returned by this plugin.

In threaded mode this is where the plugin might setup any context that needs to be shared among all output threads, such as data provided in the ConfNode.

ThreadInit

static int ThreadInit(void *init_data, int thread_id, void **thread_data)

In threaded mode this function will be called for each output thread. The arguments include the data as returned by Init in the init_data field, the thread_id of this output, and an output pointer thread_data where context data specific to this thread can be stored.

The thread_id is sequential and starts at 1, however there might be gaps, ie) 1..13,15,17.

When in threaded mode this is where the plugin would likely open a file, database connection, socket, etc.

Write

static int Write(const char *buffer, int len, void *data, void *thread_data)

The Write callback is called for each event where buffer is a formatted JSON string of the event (nul terminated string), len is the length of this string, data is the context data provided in Init and thread_data is the per-thread context data provided by ThreadInit.

ThreadDeinit

static int ThreadDeinit(void *ctx, void *thread_data)

ThreadDeinit will be called when an output thread is closed. This should free any resources allocated in ThreadInit.

TODO: This should probably return void.

Close

static int Close(void *data)

Close will be the final call into the plugin where it should cleanup any extra resources, such as those allocated in Init and provided in the data argument.

TODO: Rename to Deinit to match Init?
TODO: Probably should return void.

Actions #8

Updated by Danny Browning over 3 years ago

Would it be possible to always call ThreadInit/ThreadDeinit regardless of non-threaded or threaded mode? Plugin flow would always be Init -> ThreadInit -> Write (Repeat) -> ThreadDeinit -> Close.

Also ThreadInit should receive the previously created data from Init as a parameter.

Actions #9

Updated by Danny Browning over 3 years ago

Just realized ctx was the Init data. Maybe a rename to init_data or init_ctx?

Actions #10

Updated by Jeff Lucovsky over 3 years ago

Would it be possible to always call ThreadInit/ThreadDeinit regardless of non-threaded or threaded mode? Plugin flow would always be Init -> ThreadInit -> Write (Repeat) -> ThreadDeinit -> Close.

Yes -- this is the intended flow.

Also ThreadInit should receive the previously created data from Init as a parameter.

It does -- "The arguments include the data as returned by Init in the ctx field"

Just realized ctx was the Init data. Maybe a rename to init_data or init_ctx?

Yes --- this can be done.

Actions #11

Updated by Jason Ish over 3 years ago

Is threaded still a useful argument to have in Init? If the plugin is written in a thread safe manner (even just a lock around the same resource) then we don't need to worry about it too much. If the plugin really needs to know, it can call ConfGeBool(conf_node, "threaded").

Actions #12

Updated by Jeff Lucovsky over 3 years ago

Is threaded still a useful argument to have in Init? If the plugin is written in a thread safe manner (even just a lock around the same resource) then we don't need to worry about it too much. If the plugin really needs to know, it can call ConfGeBool(conf_node, "threaded").

I kept it in the interface to Init based on an earlier discussion we had.

You're correct -- it can decide the type of behavior to implement using ConfGetBool(..., "threaded") -- i'll update my pr

Actions #13

Updated by Jason Ish over 3 years ago

Yeah, I think it was more important when ThreadInit would not be called if unthreaded. But with ThreadInit being called no matter what, I think its less important.

Actions #14

Updated by Jeff Lucovsky over 3 years ago

Would it be possible to always call ThreadInit/ThreadDeinit regardless of non-threaded or threaded mode? Plugin flow would always be Init -> ThreadInit -> Write (Repeat) -> ThreadDeinit -> Close.

Yes -- this is the intended flow.

Hmmm ... I've polished things a bit and I don't think it'll be possible to do this since Suricata only needs calls ThreadInit/ThreadDeinit when the output logging is in threaded mode as there's additional context it stores.

Could you structure your Init function as

int Init(ConfNode *conf, bool threaded, void **init_data))
{
  .
  .
  if (!threaded)
      MyThreadInit()
  .
  .
}

int Deinit(void *init_data))
{
    if (!threaded)
       MyThreadDeinit()
    .
    .
}
Actions #15

Updated by Jason Ish over 3 years ago

Suricata may only need to call ThreadInit when it is threaded mode, but does that mean it can't also call it while not threaded mode? I guess when not in threaded mode, the support bits for passing the thread_data around are not there?

Actions #16

Updated by Danny Browning over 3 years ago

Yeah, can definitely structure plugins that way, was just seeing if we could simplify the plugin writer experience.

Can write the example plugin that way if we can't easily always call thread init/deinit.

Actions #17

Updated by Jeff Lucovsky over 3 years ago

Suricata may only need to call ThreadInit when it is threaded mode, but does that mean it can't also call it while not threaded mode? I guess when not in threaded mode, the support bits for passing the thread_data around are not there?

Yes ... The code paths are quite different in Suricata with threaded mode.

Actions #18

Updated by Jeff Lucovsky over 3 years ago

  • Status changed from In Review to Closed
Actions

Also available in: Atom PDF