Project

General

Profile

Bug #2954

Strange interaction with afpacket - high CPU usage and no packet processing

Added by Doug Burks 5 months ago. Updated 4 months ago.

Status:
New
Priority:
Normal
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

Greetings from the Security Onion community!

As you may know, Security Onion uses Suricata, netsniff-ng, and Bro. We recently started defaulting our Suricata configuration to use afpacket instead of pfring. I just noticed a strange interaction with Suricata and afpacket resulting in high CPU usage and no packet processing.

Here is what I know so far:

- If Suricata is the only sniffing process running (no netsniff-ng), then it processes traffic with normal CPU usage.

- If netsniff-ng is running and Suricata starts in afpacket mode, Suricata spikes to 100% CPU usage and stays there for a long period of time. The log reports that Suricata is initialized, but it is not processing any traffic.

- If I then try to restart Suricata immediately, it again spikes to 100% CPU usage and does not process traffic.

- After waiting a certain amount of time, I can restart Suricata and it will process traffic with normal CPU usage.

- If I start Suricata first without netsniff-ng running, Suricata will process traffic with normal CPU usage. If I then start netsniff-ng, Suricata will immediately spike to 100% CPU usage.

- netsniff-ng consumes traffic via tpacket v3.

- Bro consumes traffic from afpacket and is unaffected by all of this.

- Suricata running in pfring mode is unaffected by all of this.

- Duplicated this issue on multiple different physical boxes with Intel NICs, have not been able to duplicate in a VM yet.

- Duplicated this issue with Suricata configured for 1, 2, and 3 afpacket workers.

- Ubuntu 16.04

- Linux 4.15.0-48-generic

It almost feels as if netsniff is creating some kind of lock or changing afpacket processing in such a way as to cause Suricata to spin. After a certain amount of time, that reverts and Suricata is able to process traffic properly. Bro is unaffected by all of this, so I wonder if Suricata can workaround this condition in the same way.

Thoughts?

Thanks in advance!


Files

suricata-output.txt (191 Bytes) suricata-output.txt Doug Burks, 05/02/2019 12:52 PM
suricata-yaml-afpacket.txt (4.85 KB) suricata-yaml-afpacket.txt Doug Burks, 05/02/2019 12:52 PM
suricata-cmdline.txt (143 Bytes) suricata-cmdline.txt Doug Burks, 05/02/2019 12:52 PM
netsniff-output.txt (135 Bytes) netsniff-output.txt Doug Burks, 05/02/2019 12:52 PM
netsniff-cmdline.txt (186 Bytes) netsniff-cmdline.txt Doug Burks, 05/02/2019 12:52 PM

Related issues

Related to Feature #1954: runtime option/flag to disable hardware timestamp supportAssigned11/13/2016Actions

History

#1

Updated by Victor Julien 5 months ago

Could you attach the af-packet section from your yaml, the suricata start commandline, the netsniff-ng commandline and netsniff and suricata console (stdout/stderr) output?

#2

Updated by Doug Burks 5 months ago

It looks like adding the --no-hwtimestamp option to netsniff makes this issue go away. We'll update our scripts to use this option, so this is no longer a high priority issue for us.

However, I am curious why netsniff's default use of hardware timestamps caused an issue with Suricata (but not with Bro)?

If you're still interested in the information you requested, I can send that tomorrow.

Thanks!

#3

Updated by Doug Burks 5 months ago

I've attached the requested information.

I had been working on this issue for 2 days before creating this ticket. As it turns out, the key point from my description above was "Duplicated this issue on multiple different physical boxes with Intel NICs, have not been able to duplicate in a VM yet." Right after submitting this ticket, I started focusing on the difference between an affected physical box and an unaffected VM. I noticed on an affected physical box that the netsniff log showed "HW timestamping enabled" whereas on an unaffected VM it did not show that. So when I tried adding the --no-hwtimestamp option to netsniff-ng, Suricata was no longer affected.

#5

Updated by Andreas Herz 4 months ago

  • Assignee set to Community Ticket
  • Target version set to TBD
#6

Updated by Victor Julien 4 months ago

  • Related to Feature #1954: runtime option/flag to disable hardware timestamp support added

Also available in: Atom PDF