Project

General

Profile

Actions
Copy link

Optimization #822

closed
AS

Explore the performance impact of libhtp htp_list_t growing in size

Optimization #822: Explore the performance impact of libhtp htp_list_t growing in size

Added by Anoop Saldanha almost 13 years ago. Updated over 7 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
Effort:
Difficulty:
Label:

Description

libhtp uses a list data structure to hold various items. Explore the impact the growth of the list has on performance.

From this mail nn the oisf mailing list - "[Oisf-users] Help with 99% CPU usage", the lists have grown large and the bts indicate that the threads are spending a lot of time parsing these lists.

Files

Download all files
thread-dump01.txt (12.5 KB) thread-dump01.txt First thread dump when capturing packets Duarte Silva, 07/08/2013 06:05 AM
thread-dump02.txt (14.7 KB) thread-dump02.txt Second thread dump when capturing packets Duarte Silva, 07/08/2013 06:05 AM
thread-dump03.txt (5.45 KB) thread-dump03.txt Third thread dump when shutting down Suricata (this was taken in the same run as the second thread dump) Duarte Silva, 07/08/2013 06:05 AM

DS Updated by Duarte Silva almost 13 years ago Actions
Copy link Download all files
 #1

I have attached the last thread-dumps (it occurs less times since the detect engine optimizations). This behaviour is also noticeable when shutting down Suricata, it seems there are way to many HTTP transactions.

DS Updated by Duarte Silva almost 13 years ago Actions
Copy link
 #2

I forgot to add that when running af-packet in cluster_cpu mode, it has more loss than when running in cluster_flow. The network interface affinity settings are as follows.

           CPU0       CPU1       CPU2       CPU3
 78:  153977205          0          0          0   PCI-MSI-edge      eth1-rxtx-0
 79:          8  142687894          0          0   PCI-MSI-edge      eth1-rxtx-1
 80:          1          0  153285742          0   PCI-MSI-edge      eth1-rxtx-2
 81:          1          0          0  177757021   PCI-MSI-edge      eth1-rxtx-3
 82:          0          0          0          0   PCI-MSI-edge      eth1-event-4

I have also setup the interface ring buffers to the maximum supported.

# ethtool -g eth1
Ring parameters for eth1:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096
Current hardware settings:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             512

VJ Updated by Victor Julien almost 13 years ago Actions
Copy link
 #3

  • Status changed from New to Assigned
  • Target version changed from 2.0rc2 to 2.0beta2

VJ Updated by Victor Julien over 12 years ago Actions
Copy link
 #4

  • Target version changed from 2.0beta2 to 2.0rc1

VJ Updated by Victor Julien over 12 years ago Actions
Copy link
 #5

  • Tracker changed from Bug to Optimization

VJ Updated by Victor Julien about 12 years ago Actions
Copy link
 #6

  • Target version changed from 2.0rc1 to TBD

AH Updated by Andreas Herz over 9 years ago Actions
Copy link
 #7

  • Assignee changed from Anoop Saldanha to OISF Dev

VJ Updated by Victor Julien over 7 years ago Actions
Copy link
 #8

  • Status changed from Assigned to Closed
  • Assignee deleted (OISF Dev)
  • Target version deleted (TBD)
Actions
Copy link

Also available in: PDF Atom