Project

General

Profile

Actions

Bug #2220

open

When running on a single-CPU machine, pcap processing takes a long time

Added by Ralph Broenink about 5 years ago. Updated 10 months ago.

Status:
In Review
Priority:
Normal
Assignee:
-
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

I was attempting to batch-process a list of PCAPs, but I noticed the speed was very low on one of my worker machines. It is a virtual machine with only a single CPU. Increasing the number of CPUs resolves the issue.

I started Suricata in socket mode:
suricata --unix-socket=/tmp/suricata/socket --pidfile=/tmp/suricata/pid -S /tmp/suricata/rules

With the rules file being empty, or containing a single simple rule.

Then, I submit a lot of PCAPs to process using a very simple Python script, and interrupting it using Ctrl+C after a while:

import suricatasc
s = suricatasc.SuricataSC("/tmp/suricata/socket")
s.connect()
while True:
    s.send_command("pcap-file", {"filename": "empty.pcap", "output-dir": "/tmp/suricata/"})

On a machine with 1 CPU and 1 core, it takes almost exactly 1 second to process a single PCAP file, even if it is empty. The script above will be successful in sending two commands to the socket, so after 30s of running this script, 60 pcaps will be submitted (and 30 will still need processing).

On a machine with more CPU threads, the above works very fast.

This seems to be an issue with the threading library or so. Since my testing showed the processing taking 1 second per PCAP, I suspected a random sleep somewhere, but couldn't find it.

This issue occurs with at least Suricata 3.0, 3.2 and 4.0.


Related issues 1 (1 open0 closed)

Related to Bug #2221: Suricata batch processing slowed down by 0.2s intervalsNewOISF DevActions
Actions

Also available in: Atom PDF