Optimization #822: Explore the performance impact of libhtp htp_list_t growing in size - Suricata - Open Information Security Foundation

Actions

Copy link

Optimization #822

closed

Explore the performance impact of libhtp htp_list_t growing in size

Added by Anoop Saldanha over 11 years ago. Updated over 6 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Target version:

Effort:

Difficulty:

Label:

Description

libhtp uses a list data structure to hold various items. Explore the impact the growth of the list has on performance.

From this mail nn the oisf mailing list - "[Oisf-users] Help with 99% CPU usage", the lists have grown large and the bts indicate that the threads are spending a lot of time parsing these lists.

Files

Download all files

thread-dump01.txt (12.5 KB) thread-dump01.txt	First thread dump when capturing packets	Duarte Silva, 07/08/2013 06:05 AM
thread-dump02.txt (14.7 KB) thread-dump02.txt	Second thread dump when capturing packets	Duarte Silva, 07/08/2013 06:05 AM
thread-dump03.txt (5.45 KB) thread-dump03.txt	Third thread dump when shutting down Suricata (this was taken in the same run as the second thread dump)	Duarte Silva, 07/08/2013 06:05 AM

Actions

Copy link Download all files

Updated by Duarte Silva over 11 years ago

File thread-dump01.txt thread-dump01.txt added
File thread-dump02.txt thread-dump02.txt added
File thread-dump03.txt thread-dump03.txt added

I have attached the last thread-dumps (it occurs less times since the detect engine optimizations). This behaviour is also noticeable when shutting down Suricata, it seems there are way to many HTTP transactions.

Actions

Copy link

Updated by Duarte Silva over 11 years ago

I forgot to add that when running af-packet in cluster_cpu mode, it has more loss than when running in cluster_flow. The network interface affinity settings are as follows.

           CPU0       CPU1       CPU2       CPU3
 78:  153977205          0          0          0   PCI-MSI-edge      eth1-rxtx-0
 79:          8  142687894          0          0   PCI-MSI-edge      eth1-rxtx-1
 80:          1          0  153285742          0   PCI-MSI-edge      eth1-rxtx-2
 81:          1          0          0  177757021   PCI-MSI-edge      eth1-rxtx-3
 82:          0          0          0          0   PCI-MSI-edge      eth1-event-4

I have also setup the interface ring buffers to the maximum supported.

# ethtool -g eth1
Ring parameters for eth1:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096
Current hardware settings:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             512

Actions

Copy link