Actions
Bug #1078
closed'Not" operator (!) in Variable causes extremely slow loading of Suricata
Affected Versions:
Effort:
Difficulty:
Label:
Description
I'm running Suricata 1.4.1 on FreeBSD. I have encountered an issue when modifying the
HOME_NET variable if I want to exclude a particular IP address from a CIDR
defined network. For example, in HOME_NET I have defined 192.168.0.0/16.
If I want to EXCLUDE the 192.168.14.0 subnet, the documentation I found
said it would noted as follows:
HOME_NET: [192.168.0.0/16,!192.168.14.0/24]
If I DO NOT have the negative in my Suricata.yaml file, it loads in seconds:
-------------------------------------------------------------------- Jan 9 20:33:04 IDS_GW suricata: 9/1/2014 -- 20:33:04 - <Info> - 8641 signatures processed. 704 are IP-only rules, 3688 are inspecting packet payload, 4896 inspect application layer, 0 are decoder event only Jan 9 20:33:04 IDS_GW suricata: 9/1/2014 -- 20:33:04 - <Info> - building signature grouping structure, stage 1: adding signatures to signature source addresses... complete Jan 9 20:33:04 IDS_GW suricata: 9/1/2014 -- 20:33:04 - <Info> - building signature grouping structure, stage 2: building source address list... complete Jan 9 20:33:11 IDS_GW suricata: 9/1/2014 -- 20:33:11 - <Info> - building signature grouping structure, stage 3: building destination address lists... complete ---------------------------------------------------------------------
When I add the negative to the variable, it takes over
40 minutes for Suricata to load and start inspecting traffic.
From the logs, I can tell it hangs after stage 1. Notice the times:
--------------------------------------------------------------------- 9/1/2014 -- 20:23:29 - <Info> - building signature grouping structure, stage 1: adding signatures to signature source addresses... complete 9/1/2014 -- 20:53:39 - <Info> - building signature grouping structure, stage 2: building source address list... complete 9/1/2014 -- 21:10:09 - <Info> - building signature grouping structure, stage 3: building destination address lists... complete ---------------------------------------------------------------------
This bug was independently verified on 2.0Beta2 as well.
Updated by Victor Julien over 10 years ago
- Description updated (diff)
- Target version changed from 1.4.1 to 2.0rc2
Updated by Victor Julien about 10 years ago
- Target version changed from 2.0rc2 to 3.0RC2
Likely related to the address grouping code. Hoping to replace that in 2.1.
Updated by Victor Julien over 8 years ago
- Target version changed from 3.0RC2 to 70
Updated by Victor Julien about 8 years ago
- Status changed from New to Assigned
- Assignee set to Victor Julien
I expect this to be resolved in my detection engine rewrite work: https://github.com/inliniac/suricata/pull/1811
Updated by Peter Manev about 8 years ago
It does not seem to resolve it. (This is Suricata version 3.0dev (rev 11bd9f1))
Interestingly enough in my case it is the trojan.rules that take the longest time when i have vegetated net variable inside HOME_NET example-
[26082] 29/1/2016 -- 03:10:33 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/local.rules [26082] 29/1/2016 -- 03:10:33 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/attack_response.rules [26082] 29/1/2016 -- 03:10:34 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/exploit.rules [26082] 29/1/2016 -- 03:10:42 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/ftp.rules [26082] 29/1/2016 -- 03:10:43 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/scan.rules [26082] 29/1/2016 -- 03:10:45 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/shellcode.rules [26082] 29/1/2016 -- 03:10:46 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/smtp.rules [26082] 29/1/2016 -- 03:10:46 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/snmp.rules [26082] 29/1/2016 -- 03:10:46 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/sql.rules [26082] 29/1/2016 -- 03:10:48 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/telnet.rules [26082] 29/1/2016 -- 03:10:48 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/trojan.rules [26082] 29/1/2016 -- 03:12:40 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/user_agents.rules [26082] 29/1/2016 -- 03:12:41 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/web_server.rules
As opposed to when I don't have negated net variables inside HOME_NET:
[44479] 29/1/2016 -- 04:19:42 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/local.rules [44479] 29/1/2016 -- 04:19:42 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/attack_response.rules [44479] 29/1/2016 -- 04:19:42 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/exploit.rules [44479] 29/1/2016 -- 04:19:42 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/ftp.rules [44479] 29/1/2016 -- 04:19:42 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/scan.rules [44479] 29/1/2016 -- 04:19:43 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/shellcode.rules [44479] 29/1/2016 -- 04:19:43 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/smtp.rules [44479] 29/1/2016 -- 04:19:43 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/snmp.rules [44479] 29/1/2016 -- 04:19:43 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/sql.rules [44479] 29/1/2016 -- 04:19:43 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/telnet.rules [44479] 29/1/2016 -- 04:19:43 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/trojan.rules [44479] 29/1/2016 -- 04:19:50 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/user_agents.rules [44479] 29/1/2016 -- 04:19:50 - (detect.c:417) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/web_server.rules
Updated by Victor Julien about 8 years ago
Ah I see, so it's the actual parsing that is slow. Will look into it.
Updated by Peter Manev about 8 years ago
Just looking at perf top while loading - it seems it spends a lot of time in the function
DetectAddressCmpIPv4
before moving forward - whereas if there are no negated variables in HOME_NET - it passes through there in a few seconds.
Updated by Victor Julien almost 8 years ago
- Status changed from Assigned to Closed
- Target version changed from 70 to 3.1rc1
Looks like this has been solved with the detection engine rewrite.
Actions