Project

General

Profile

Bug #2221

Suricata batch processing slowed down by 0.2s intervals

Added by Ralph Broenink about 1 year ago. Updated about 1 year ago.

Status:
New
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:

Description

I was attempting to batch-process a list of PCAPs, (see also issue #2220), I noticed that while submitting PCAPs to Suricata, processing seemed very fast, but after this was done, processing was limited to 5 PCAPs/second.

I started Suricata in socket mode:
suricata --unix-socket=/tmp/suricata/socket --pidfile=/tmp/suricata/pid -S /tmp/suricata/rules

With the rules file being empty, or containing a single simple rule. Then, I submit a lot of PCAPs to process using a very simple Python script, and interrupting it using Ctrl+C after a while:

import suricatasc
s = suricatasc.SuricataSC("/tmp/suricata/socket")
s.connect()
while True:
    s.send_command("pcap-file", {"filename": "empty.pcap", "output-dir": "/tmp/suricata/"})

Processing of Suricata slowed down to at most 5 PCAPs/second after Ctrl+C'ing the above script. This seems to be due to the fact that after each PCAP processing, Suricata reads messages for 0.2s from the Socket interface. After not having received any messages, it continues processing. This seems to be caused by the function UnixMain of unix-manager.c, where the 200ms timeout is specified.

I feel the polling of the socket should probably be done in a separate thread. This would accomplish two things:
  • PCAP batch processing is not slowing down when there are no messages served on the socket
  • Clients sending messages on the socket do not have to wait for Suricata when it is processing a large PCAP

Note that the issue is resolved by, after submitting all PCAPs, continuing to send messages on the socket.

This issue occurs with at least Suricata 3.2 and 4.0.

History

#1 Updated by Victor Julien about 1 year ago

  • Assignee set to Eric Leblond
  • Priority changed from High to Normal

Eric any thoughts?

#2 Updated by Andreas Herz about 1 year ago

  • Target version set to TBD

Also available in: Atom PDF