Project

General

Profile

Actions
Copy link

Optimization #1770

open
PM LS

Suricata takes very long time to start using hyperscan and large/custom detect settings

Optimization #1770: Suricata takes very long time to start using hyperscan and large/custom detect settings

Added by Peter Manev over 10 years ago. Updated 5 months ago.

Status:
Feedback
Priority:
Normal
Assignee:
Lukas Sismis
Target version:
9.0.0-beta1
Effort:
Difficulty:
Label:

Description

Using 3.1dev (rev b92a08b) with the following settings -

detect:
  profile: custom
  custom-values:
    toclient-groups: 600
    toserver-groups: 800
  sgh-mpm-context: full

and 

mpm-algo: hs

it takes very long time for Suricata to start as opposed to using the same settings but with 
mpm-algo: ac-ks


  profile: low/medium/high

are working fine though.

PM Updated by Peter Manev over 10 years ago Actions
Copy link
 #1 

mpm-algo: hs
detect.profile = high
detect.sgh-mpm-context: full

also takes much longer (though not as long as detect.profile = custom) than mpm-algo: ac/mpm-algo: ac-ks

AH Updated by Andreas Herz about 10 years ago Actions
Copy link
 #2

  • Assignee set to Anonymous
  • Target version set to TBD

AH Updated by Andreas Herz over 7 years ago Actions
Copy link
 #3

  • Assignee set to Community Ticket

AH Updated by Andreas Herz over 7 years ago Actions
Copy link
 #4

Is this still an issue?

PM Updated by Peter Manev over 7 years ago Actions
Copy link
 #5

  • Status changed from New to Closed

With Hyperscan yes (though not as long as initially reported) - 

suricata -i eno1 --set "detect.profile = high"  --set "detect.sgh-mpm-context = full" 
[12576] 30/7/2019 -- 15:10:31 - (suricata.c:1071) <Notice> (LogVersion) -- This is Suricata version 5.0.0-dev (3a912446a 2019-07-22) running in SYSTEM mode
....

[12576] 30/7/2019 -- 15:13:58 - (tm-threads.c:2145) <Notice> (TmThreadWaitOnThreadInit) -- all 112 packet processing threads, 10 management threads initialized, engine started.

PM Updated by Peter Manev over 7 years ago Actions
Copy link
 #6

  • Status changed from Closed to Feedback

AH Updated by Andreas Herz over 7 years ago Actions
Copy link
 #7

Does this increase with the amount of rules as well?

PM Updated by Peter Manev over 7 years ago Actions
Copy link
 #8

Yes - if you specify 0 rules it will load faster.

AH Updated by Andreas Herz over 7 years ago Actions
Copy link
 #9

what system is this? I also see quite a long load time around 30-40s sometimes but with over 35k rules.

PM Updated by Peter Manev about 7 years ago Actions
Copy link
 #10

Any Debian and Ubuntu with the following settings in yaml -

detect:
  profile: custom
  custom-values:
    toclient-groups: 600
    toserver-groups: 800
  sgh-mpm-context: full

mpm-algo: hs

It is not seconds but minutes it needs.
Andreas - are you able to reproduce this ?

AH Updated by Andreas Herz about 7 years ago Actions
Copy link
 #11

I meant more in regards to hardware :)

I see a rather big amount of threads, can you check if it changes if you change the threads amount?

PM Updated by Peter Manev about 7 years ago Actions
Copy link
 #12

No , not related to HW/number of threads in my case.
Can you please confirm ?

AH Updated by Andreas Herz about 7 years ago Actions
Copy link
 #13

I did check again in detail and confirm that it's taking very long:

22/8/2019 -- 07:45:16 - <Notice> - This is Suricata version 4.1.4 RELEASE
22/8/2019 -- 08:10:01 - <Notice> - all 16 packet processing threads, 6 management threads initialized, engine started.

Seems to increase a lot with more rules :)

PM Updated by Peter Manev about 7 years ago Actions
Copy link
 #14

Thank you for confirming.
I think the combination

detect:
  profile: custom
  custom-values:
    toclient-groups: 600
    toserver-groups: 800
  sgh-mpm-context: full

is excessive but even with profile: high when using mpm-algo hyperscan the load times are high - good few minutes with a full ruleset.

PA Updated by Philippe Antoine over 1 year ago Actions
Copy link
 #15

  • Tracker changed from Bug to Optimization

JI Updated by Jason Ish over 1 year ago Actions
Copy link
 #16

Is this still an issue, or does the caching help with this case (@Lukas Sismis ?)

LS Updated by Lukas Sismis 5 months ago Actions
Copy link
 #17

  • Assignee changed from Community Ticket to Lukas Sismis
  • Target version changed from TBD to 9.0.0-beta1

Assigning to myself to reverify. Thanks @Philippe Antoine

Actions
Copy link

Also available in: PDF Atom