Project

General

Profile

Actions

Bug #2221

open

Suricata batch processing slowed down by 0.2s intervals

Added by Ralph Broenink about 5 years ago. Updated almost 3 years ago.

Status:
New
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

I was attempting to batch-process a list of PCAPs, (see also issue #2220), I noticed that while submitting PCAPs to Suricata, processing seemed very fast, but after this was done, processing was limited to 5 PCAPs/second.

I started Suricata in socket mode:
suricata --unix-socket=/tmp/suricata/socket --pidfile=/tmp/suricata/pid -S /tmp/suricata/rules

With the rules file being empty, or containing a single simple rule. Then, I submit a lot of PCAPs to process using a very simple Python script, and interrupting it using Ctrl+C after a while:

import suricatasc
s = suricatasc.SuricataSC("/tmp/suricata/socket")
s.connect()
while True:
    s.send_command("pcap-file", {"filename": "empty.pcap", "output-dir": "/tmp/suricata/"})

Processing of Suricata slowed down to at most 5 PCAPs/second after Ctrl+C'ing the above script. This seems to be due to the fact that after each PCAP processing, Suricata reads messages for 0.2s from the Socket interface. After not having received any messages, it continues processing. This seems to be caused by the function UnixMain of unix-manager.c, where the 200ms timeout is specified.

I feel the polling of the socket should probably be done in a separate thread. This would accomplish two things:
  • PCAP batch processing is not slowing down when there are no messages served on the socket
  • Clients sending messages on the socket do not have to wait for Suricata when it is processing a large PCAP

Note that the issue is resolved by, after submitting all PCAPs, continuing to send messages on the socket.

This issue occurs with at least Suricata 3.2 and 4.0.


Related issues 1 (1 open0 closed)

Related to Bug #2220: When running on a single-CPU machine, pcap processing takes a long timeIn ReviewActions
Actions #1

Updated by Victor Julien about 5 years ago

  • Assignee set to Eric Leblond
  • Priority changed from High to Normal

Eric any thoughts?

Actions #2

Updated by Andreas Herz about 5 years ago

  • Target version set to TBD
Actions #3

Updated by Andreas Herz over 3 years ago

Do we have any thoughts on the proposed solutions?

Actions #4

Updated by Andreas Herz over 3 years ago

  • Related to Bug #2220: When running on a single-CPU machine, pcap processing takes a long time added
Actions #5

Updated by Victor Julien almost 3 years ago

  • Assignee changed from Eric Leblond to OISF Dev
Actions

Also available in: Atom PDF