Project

General

Profile

Actions
Copy link

Feature #352

closed
EL

Switching to message queuing system for output

Feature #352: Switching to message queuing system for output

Added by Eric Leblond over 14 years ago. Updated over 6 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
Effort:
Difficulty:
Label:

Description

Actual threading model consider output modules as standard functions which are run synchronously in the life of a packet inside suricata. But logging functions involve I/O on disk or on the network. They thus can be really time expensive.
There is a potential performance issue there because, the building the alert message trigger the locking of ressources like flow. It will thus be interesting to switch:
  • to asynchrounous I/O operation OR
  • to a message queuing system where an alert structure build from a copy of information is sent to the output modules.

DY Updated by delta yeh over 14 years ago Actions
Copy link
 #1

ZeroMQ maybe a candidate

VJ Updated by Victor Julien over 14 years ago Actions
Copy link
 #2

  • Status changed from New to Assigned
  • Assignee set to Eric Leblond
  • Target version set to 1.3beta2

Message queueing a good idea. Lets implement this for 1.3.

Can you come up with a higher level design?

VJ Updated by Victor Julien about 14 years ago Actions
Copy link
 #3

  • Target version changed from 1.3beta2 to 1.4

VJ Updated by Victor Julien over 13 years ago Actions
Copy link
 #4

  • Target version changed from 1.4 to 2.0rc2

Lets discuss this in Amsterdam next month.

VJ Updated by Victor Julien almost 13 years ago Actions
Copy link
 #5

  • Target version changed from 2.0rc2 to TBD

VJ Updated by Victor Julien over 12 years ago Actions
Copy link
 #6

  • Target version changed from TBD to 3.0RC2

VJ Updated by Victor Julien over 11 years ago Actions
Copy link
 #7

  • Target version changed from 3.0RC2 to 70

VJ Updated by Victor Julien about 10 years ago Actions
Copy link
 #8

We now support redis. Probably good enough for now?

JM Updated by James Mistry almost 9 years ago Actions
Copy link
 #9

I'm interested in this as a way of using Suricata as a pre-filter for traffic/parse results to be passed to external software components - that are very loosely-coupled with Suricata - for more advanced/complex analysis in real-time.

Some things that would make this feature work well for this use case:

  • Messages are serialised to a format that supports explicit/formal schemas, ideally that can be evolved in a backwards-compatible way. For example, Apache Avro or Protobufs.
  • Output flow/packet parse results with alerts, as well as raw packet/session data, in a way that is cheap to de-serialise (e.g. avoid Base64).
  • Control topics associated with messages within the signature.

I'm happy to have this feature assigned to me if that helps.

VJ Updated by Victor Julien almost 8 years ago Actions
Copy link
 #10

  • Status changed from Assigned to New
  • Assignee changed from Eric Leblond to Anonymous
  • Target version changed from 70 to TBD
  • Effort set to medium
  • Difficulty set to high

Setting difficulty to high as this could severely impact perf if done wrong.

AH Updated by Andreas Herz about 7 years ago Actions
Copy link
 #11

  • Assignee set to Community Ticket

JI Updated by Jason Ish over 6 years ago Actions
Copy link
 #12

I propose we close/reject this issue as its something we often agree is not to be part of Suricata. Our queue is the disk, or the socket or named pipe (and whatever is at the end of that). And we have other issues to address specific performance issues of the logging output.

VJ Updated by Victor Julien over 6 years ago Actions
Copy link
 #13

  • Status changed from New to Closed
  • Assignee deleted (Community Ticket)
  • Target version deleted (TBD)
  • Effort deleted (medium)
  • Difficulty deleted (high)

Agreed, thanks Jason.

Actions
Copy link

Also available in: PDF Atom