Project

General

Profile

Actions
Copy link

Bug #1078

closed
MM VJ

'Not" operator (!) in Variable causes extremely slow loading of Suricata

Bug #1078: 'Not" operator (!) in Variable causes extremely slow loading of Suricata

Added by Michael McAndrews over 12 years ago. Updated almost 10 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Victor Julien
Target version:
3.1rc1
Affected Versions:
Effort:
Difficulty:
Label:

Description

I'm running Suricata 1.4.1 on FreeBSD. I have encountered an issue when modifying the
HOME_NET variable if I want to exclude a particular IP address from a CIDR
defined network. For example, in HOME_NET I have defined 192.168.0.0/16.
If I want to EXCLUDE the 192.168.14.0 subnet, the documentation I found
said it would noted as follows:

HOME_NET:     [192.168.0.0/16,!192.168.14.0/24]

If I DO NOT have the negative in my Suricata.yaml file, it loads in seconds:

--------------------------------------------------------------------
Jan  9 20:33:04 IDS_GW suricata: 9/1/2014 -- 20:33:04 - <Info> - 8641
signatures processed. 704 are IP-only rules, 3688 are inspecting packet
payload, 4896 inspect application layer, 0 are decoder event only
Jan  9 20:33:04 IDS_GW suricata: 9/1/2014 -- 20:33:04 - <Info> -
building signature grouping structure, stage 1: adding signatures to
signature source addresses... complete
Jan  9 20:33:04 IDS_GW suricata: 9/1/2014 -- 20:33:04 - <Info> -
building signature grouping structure, stage 2: building source address
list... complete
Jan  9 20:33:11 IDS_GW suricata: 9/1/2014 -- 20:33:11 - <Info> -
building signature grouping structure, stage 3: building destination
address lists... complete
---------------------------------------------------------------------

When I add the negative to the variable, it takes over
40 minutes for Suricata to load and start inspecting traffic.

From the logs, I can tell it hangs after stage 1. Notice the times:

---------------------------------------------------------------------
9/1/2014 -- 20:23:29 - <Info> - building signature grouping structure,
stage 1: adding signatures to signature source addresses... complete
9/1/2014 -- 20:53:39 - <Info> - building signature grouping structure,
stage 2: building source address list... complete
9/1/2014 -- 21:10:09 - <Info> - building signature grouping structure,
stage 3: building destination address lists... complete
---------------------------------------------------------------------

This bug was independently verified on 2.0Beta2 as well.

VJ Updated by Victor Julien about 12 years ago Actions
Copy link
 #1

  • Description updated (diff)
  • Target version changed from 1.4.1 to 2.0rc2

VJ Updated by Victor Julien about 12 years ago Actions
Copy link
 #2

  • Target version changed from 2.0rc2 to 3.0RC2

Likely related to the address grouping code. Hoping to replace that in 2.1.

VJ Updated by Victor Julien over 10 years ago Actions
Copy link
 #3

  • Target version changed from 3.0RC2 to 70

VJ Updated by Victor Julien about 10 years ago Actions
Copy link
 #4

  • Status changed from New to Assigned
  • Assignee set to Victor Julien

I expect this to be resolved in my detection engine rewrite work: https://github.com/inliniac/suricata/pull/1811

PM Updated by Peter Manev about 10 years ago Actions
Copy link
 #5

It does not seem to resolve it. (This is Suricata version 3.0dev (rev 11bd9f1))

Interestingly enough in my case it is the trojan.rules that take the longest time when i have vegetated net variable inside HOME_NET example- 

[26082] 29/1/2016 -- 03:10:33 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/local.rules
[26082] 29/1/2016 -- 03:10:33 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/attack_response.rules
[26082] 29/1/2016 -- 03:10:34 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/exploit.rules
[26082] 29/1/2016 -- 03:10:42 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/ftp.rules
[26082] 29/1/2016 -- 03:10:43 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/scan.rules
[26082] 29/1/2016 -- 03:10:45 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/shellcode.rules
[26082] 29/1/2016 -- 03:10:46 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/smtp.rules
[26082] 29/1/2016 -- 03:10:46 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/snmp.rules
[26082] 29/1/2016 -- 03:10:46 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/sql.rules
[26082] 29/1/2016 -- 03:10:48 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/telnet.rules
[26082] 29/1/2016 -- 03:10:48 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/trojan.rules
[26082] 29/1/2016 -- 03:12:40 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/user_agents.rules
[26082] 29/1/2016 -- 03:12:41 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/web_server.rules

As opposed to when I don't have negated net variables inside HOME_NET:

[44479] 29/1/2016 -- 04:19:42 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/local.rules
[44479] 29/1/2016 -- 04:19:42 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/attack_response.rules
[44479] 29/1/2016 -- 04:19:42 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/exploit.rules
[44479] 29/1/2016 -- 04:19:42 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/ftp.rules
[44479] 29/1/2016 -- 04:19:42 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/scan.rules
[44479] 29/1/2016 -- 04:19:43 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/shellcode.rules
[44479] 29/1/2016 -- 04:19:43 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/smtp.rules
[44479] 29/1/2016 -- 04:19:43 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/snmp.rules
[44479] 29/1/2016 -- 04:19:43 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/sql.rules
[44479] 29/1/2016 -- 04:19:43 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/telnet.rules
[44479] 29/1/2016 -- 04:19:43 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/trojan.rules
[44479] 29/1/2016 -- 04:19:50 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/user_agents.rules
[44479] 29/1/2016 -- 04:19:50 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/web_server.rules

VJ Updated by Victor Julien about 10 years ago Actions
Copy link
 #6

Ah I see, so it's the actual parsing that is slow. Will look into it.

PM Updated by Peter Manev about 10 years ago Actions
Copy link
 #7

Just looking at perf top while loading - it seems it spends a lot of time in the function 

DetectAddressCmpIPv4

before moving forward - whereas if there are no negated variables in HOME_NET - it passes through there in a few seconds.

VJ Updated by Victor Julien almost 10 years ago Actions
Copy link
 #8

  • Status changed from Assigned to Closed
  • Target version changed from 70 to 3.1rc1

Looks like this has been solved with the detection engine rewrite.

Actions
Copy link

Also available in: PDF Atom