Project

General

Profile

Actions

Bug #7048

closed

af-packet: failure to start up on many threads plus high load

Added by Victor Julien 6 months ago. Updated 6 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

Suricata can fail to start up with this error:

Error: threads: thread "W#01-ens6f0" failed to start in time: flags 0003 [TmThreadWaitOnThreadRunning:tm-threads.c:1840]

This happens when there are many threads (>100) and they have a large ring-size in af-packet and the network is very loaded. The start up and AFPSynchronizeStart appear to take too much time, and the TmThreadWaitOnThreadRunning function fails with the error above. This timer allows for 60s for the threads to start up and synchronize.

Almost all time is spent in the kernel, it seems:

  96.16%  [kernel.kallsyms]                                                                                                                                                                                                                                                               
   2.37%  libc.so.6                                                                                                                                                                                                                                                                       
   1.37%  suricata                                                                                                                                                                                                                                                                        
   0.05%  libhs.so.5.4.0                                                                                                                                                                                                                                                                  
   0.03%  [vdso]                                                                                                                                                                                                                                                                          
   0.01%  ld-linux-x86-64.so.2                                                                                                                                                                                                                                                            
   0.00%  libpcre2-8.so.0.10.4                                                                                                                                                                                                                                                            
   0.00%  libjansson.so.4.13.0                                                                                                                                                                                                                                                            
   0.00%  libstdc++.so.6.0.30                                                                                                                                                                                                                                                             
   0.00%  libyaml-0.so.2.0.6


Subtasks 1 (0 open1 closed)

Bug #7050: af-packet: failure to start up on many threads plus high load (7.0.x backport)ClosedVictor JulienActions

Related issues 1 (1 open0 closed)

Related to Suricata - Optimization #7272: af-packet: improve startup timeIn ReviewVictor JulienActions
Actions #1

Updated by Victor Julien 6 months ago

  • Status changed from New to In Review
  • Assignee changed from OISF Dev to Victor Julien
  • Target version changed from TBD to 8.0.0-beta1
  • Label Needs backport to 7.0 added

Reviewing the af-packet logic doesn't show me anything being wrong, just that it takes a lot of time to start the threads sequentially. So the PR https://github.com/OISF/suricata/pull/11169 increases the time based on the number of threads.

Actions #2

Updated by OISF Ticketbot 6 months ago

  • Subtask #7050 added
Actions #3

Updated by OISF Ticketbot 6 months ago

  • Label deleted (Needs backport to 7.0)
Actions #4

Updated by Victor Julien 6 months ago

  • Status changed from In Review to Resolved
Actions #5

Updated by Victor Julien 6 months ago

  • Status changed from Resolved to Closed
Actions #6

Updated by Victor Julien about 2 months ago

Actions

Also available in: Atom PDF