This might be helpful for high performance setups that use redis, since it would help to scale the bottleneck with the single redis pipeline

Here are some draft notes on the life cycle of a plugin with this PR that I'm hoping we can get some more feedback on.

Callbacks¶

• Init
• Close (TODO: Rename to Deinit?)
• ThreadInit
• ThreadDeinit
• Write

Init¶

static int Init(ConfNode *conf, bool threaded, void **data)

The Init function is the first callback called. The arguments include the ConfNode for the Eve output using the plugin, a flag to tell if the Eve output is multi-threaded or not and an output pointer to provide any data to future callbacks.

As threading of the output is not under the control of the plugin, the plugin needs to decide if it can handle being threaded or not. If the plugin cannot handle being threaded it should return from Init with an error code -1.

Or if the plugin cannot run unthreaded, it could return an error code if not threaded.

Ideally output plugins can handle both cases.

In non-threaded mode this is where the plugin might open a file, a database connection, a socket, etc. The next call into the plugin will the be the Write callback along with the data returned by this plugin.

In threaded mode this is where the plugin might setup any context that needs to be shared among all output threads, such as data provided in the ConfNode.

ThreadInit¶

static int ThreadInit(void *init_data, int thread_id, void **thread_data)

In threaded mode this function will be called for each output thread. The arguments include the data as returned by Init in the init_data field, the thread_id of this output, and an output pointer thread_data where context data specific to this thread can be stored.

The thread_id is sequential and starts at 1, however there might be gaps, ie) 1..13,15,17.

When in threaded mode this is where the plugin would likely open a file, database connection, socket, etc.

Write¶

static int Write(const char *buffer, int len, void *data, void *thread_data)

The Write callback is called for each event where buffer is a formatted JSON string of the event (nul terminated string), len is the length of this string, data is the context data provided in Init and thread_data is the per-thread context data provided by ThreadInit.

ThreadDeinit¶

static int ThreadDeinit(void *ctx, void *thread_data)

ThreadDeinit will be called when an output thread is closed. This should free any resources allocated in ThreadInit.

TODO: This should probably return void.

Close¶

static int Close(void *data)

Close will be the final call into the plugin where it should cleanup any extra resources, such as those allocated in Init and provided in the data argument.

TODO: Rename to Deinit to match Init?
TODO: Probably should return void.

Would it be possible to always call ThreadInit/ThreadDeinit regardless of non-threaded or threaded mode? Plugin flow would always be Init -> ThreadInit -> Write (Repeat) -> ThreadDeinit -> Close.

Also ThreadInit should receive the previously created data from Init as a parameter.

Just realized ctx was the Init data. Maybe a rename to init_data or init_ctx?

Would it be possible to always call ThreadInit/ThreadDeinit regardless of non-threaded or threaded mode? Plugin flow would always be Init -> ThreadInit -> Write (Repeat) -> ThreadDeinit -> Close.

Yes -- this is the intended flow.

Also ThreadInit should receive the previously created data from Init as a parameter.

It does -- "The arguments include the data as returned by Init in the ctx field"

Just realized ctx was the Init data. Maybe a rename to init_data or init_ctx?

Yes --- this can be done.

Is threaded still a useful argument to have in Init? If the plugin is written in a thread safe manner (even just a lock around the same resource) then we don't need to worry about it too much. If the plugin really needs to know, it can call ConfGeBool(conf_node, "threaded").

Is threaded still a useful argument to have in Init? If the plugin is written in a thread safe manner (even just a lock around the same resource) then we don't need to worry about it too much. If the plugin really needs to know, it can call ConfGeBool(conf_node, "threaded").

I kept it in the interface to Init based on an earlier discussion we had.

You're correct -- it can decide the type of behavior to implement using ConfGetBool(..., "threaded") -- i'll update my pr

Yeah, I think it was more important when ThreadInit would not be called if unthreaded. But with ThreadInit being called no matter what, I think its less important.

Would it be possible to always call ThreadInit/ThreadDeinit regardless of non-threaded or threaded mode? Plugin flow would always be Init -> ThreadInit -> Write (Repeat) -> ThreadDeinit -> Close.

Yes -- this is the intended flow.

Hmmm ... I've polished things a bit and I don't think it'll be possible to do this since Suricata only needs calls ThreadInit/ThreadDeinit when the output logging is in threaded mode as there's additional context it stores.

Could you structure your Init function as

int Init(ConfNode *conf, bool threaded, void **init_data))
{
  .
  .
  if (!threaded)
      MyThreadInit()
  .
  .
}

int Deinit(void *init_data))
{
    if (!threaded)
       MyThreadDeinit()
    .
    .
}

Suricata may only need to call ThreadInit when it is threaded mode, but does that mean it can't also call it while not threaded mode? I guess when not in threaded mode, the support bits for passing the thread_data around are not there?

Yeah, can definitely structure plugins that way, was just seeing if we could simplify the plugin writer experience.

Can write the example plugin that way if we can't easily always call thread init/deinit.

Suricata may only need to call ThreadInit when it is threaded mode, but does that mean it can't also call it while not threaded mode? I guess when not in threaded mode, the support bits for passing the thread_data around are not there?

Yes ... The code paths are quite different in Suricata with threaded mode.