Actions
Bug #2908
closedip only rules cause suricata to take 17 minutes to start
Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
Affected Versions:
Effort:
Difficulty:
Label:
Description
We are trying to run with 300+ CIDRs in our home net with about 50k rules (et pro and others). Suricata 4.1.2 & 4.1.3 takes about 17 minutes between when we get the rules loaded and when we get the af packet threads are now listening print out. Using gperftools it says that over 97% of the time is spent in IPOnlyCIDRItemInsertReal. Are we doing something wrong?
A quick glance at the code makes it looks like a link list insertion sort is being used to sort these 300+ cidrs for every single rule?
Seems like some possible solutions- We could presort our config from max netmask to smallest, so the sort would be O(n). We have our ipv6 ips last in the list, so I bet currently we are worst case near O(n^2)
- The code could switch to a qsort instead of link list insertion sort
- The code could cache the list
Any help would be great!
Updated by Victor Julien about 5 years ago
So this is a normal ruleset like ET used with a large HOME_NET? Are you able to share your HOME_NET setting, or create a fake one that has the same effect?
Updated by Andy Wick about 5 years ago
To make it easy to repo I switched to just the full et pro rule set and nothing else. Which outputs
45259 signatures processed. 1225 are IP-only rules, 17090 are inspecting packet payload, 31407 inspect application layer, 0 are decoder event only
I changed LoadSignatures to do a exit instead of returning for all times below.
So running with time /nids/suricata/bin/suricata -c /nids/suricata/etc/suricata.yaml --af-packet -v Our original ip list: real 29m15.512s user 28m49.707s sys 0m25.756s Then I sorted the list with: sort -t/ -k2 -n -r real 11m20.801s user 11m18.115s sys 0m2.682s
So sorting helped a lot. I tried to make a sorted fake list, which isn't quite as slow, but still almost 8m.
real 7m58.549s user 7m56.561s sys 0m1.986s "[1111:2222:3333:7fff::a:1/128,1111:2222:3333:7eff::a:1/128,1212:2222:3333:7fff::a:1/128,1212:2222:3333:7eff::a:1/128,1313:2222:3333:ff::a:2/128,1313:2222:3333:ff::a:1/128,1313:2222:3333:7cff::a:1/128,1111:2222:3333:7ffe::/127,1111:2222:3333:7ffc::/127,1212:2222:3333:7ffe::/127,1212:2222:3333:7ffc::/127,1414:2222:f024:1ff::/64,1111:2222:3333::/64,1111:2222:3333:7f02::/64,1414:2222:efca:7ffc::/64,1414:2222:efca:7f02::/64,1414:2222:efca:7eff::/64,1414:2222:efca:1::/64,1212:2222:3333:7f02::/64,1212:2222:3333:2::/64,1212:2222:3333:1::/64,1414:2222:ef98:7ffd::/64,1414:2222:ef98:7eff::/64,1414:2222:ef98:7bff::/64,1414:2222:ef98:7808::/64,1414:2222:ef98:7801::/64,1414:2222:ef98:110::/64,1414:2222:ef98:10c::/64,1515:2222:3333:8::/64,1515:2222:3333:7ffd::/64,1515:2222:3333:7eff::/64,1515:2222:3333:7dff::/64,1515:2222:3333:7d00::/64,1616:2222:3333:7ffd::/64,1616:2222:3333:7eff::/64,1616:2222:3333:7dff::/64,1616:2222:3333:7d00::/64,1616:2222:3333:7801::/64,1717:2222:3333:7eff::/64,1717:2222:3333:7dff::/64,1717:2222:3333:7d00::/64,1717:2222:3333:7806::/64,1717:2222:3333:7801::/64,1717:2222:3333:1fc::/64,1717:2222:3333:14::/64,1313:2222:3333:fe::/64,1313:2222:3333:8::/64,1313:2222:3333:7ffd::/64,1313:2222:3333:7eff::/64,1313:2222:3333:7dff::/64,1313:2222:3333:7d00::/64,1313:2222:3333:7cfe::/64,1313:2222:3333:7cfd::/64,1313:2222:3333:7800::/64,1313:2222:3333:11a::/64,1313:2222:3333:119::/64,1818:2222:3333:1007::/64,1919:2222:3333:a::/64,1111:2222:3333:7f00::/63,1414:2222:efca:7ffe::/63,1414:2222:efca:7f00::/63,1414:2222:efca:2::/63,1212:2222:3333:7f00::/63,1414:2222:ef98:7ffe::/63,1414:2222:ef98:7f00::/63,1414:2222:ef98:78fe::/63,1414:2222:ef98:7802::/63,1515:2222:3333:7ffe::/63,1515:2222:3333:7f00::/63,1616:2222:3333:7ffe::/63,1616:2222:3333:78fe::/63,1616:2222:3333:7802::/63,1717:2222:3333:7ffe::/63,1717:2222:3333:7f00::/63,1717:2222:3333:7804::/63,1717:2222:3333:7802::/63,1717:2222:3333:1fe::/63,1717:2222:3333:1f4::/63,1313:2222:3333:fc::/63,1313:2222:3333:a::/63,1313:2222:3333:7ffe::/63,1313:2222:3333:7f00::/63,1313:2222:3333:7c08::/63,1313:2222:3333:1fe::/63,1414:2222:ef98:fc::/62,1414:2222:ef98:7cfc::/62,1414:2222:ef98:7c00::/62,1414:2222:ef98:7b00::/62,1414:2222:ef98:7804::/62,1414:2222:ef98:1fc::/62,1414:2222:ef98:108::/62,1515:2222:3333:fc::/62,1616:2222:3333:fc::/62,1616:2222:3333:7f00::/62,1616:2222:3333:18::/62,1717:2222:3333:1f0::/62,1313:2222:3333:7804::/62,1313:2222:3333:110::/62,1414:2222:ef98::/61,1414:2222:ef98:100::/61,1515:2222:3333::/61,1616:2222:3333:10::/61,1717:2222:3333:100::/61,1313:2222:3333::/61,1313:2222:3333:7c00::/61,1313:2222:3333:7808::/61,1616:2222:3333::/60,1313:2222:3333:7810::/60,1313:2222:3333:100::/60,1313:2222:3333:7820::/59,1313:2222:3333:7840::/58,1313:2222:3333:7880::/57,1414:2222:ef98:7d00::/56,1313:2222:3333:7900::/56,1313:2222:3333:7a00::/55,1515:2222:3333:8000::/49,1616:2222:3333:8000::/49,1414:2222:efc4::/48,1414:2222:efc2::/48,1414:2222:ef84::/48,2020:2222:efba::/48,2020:2222:efb8::/48,2020:2222:efaf::/48,2020:2222:efa8::/48,2020:2222:efa7::/48,2020:2222:efa5::/48,2020:2222:ef96::/48,2121:2222:fbff::/48,2121:2222:eff5::/48,2121:2222:efeb::/48,2121:2222:efd8::/48,2121:2222:efad::/48,2121:2222:ef99::/48,2121:2222:ef94::/48,2121:2222:ef8f::/48,2121:2222:ef83::/48,2121:2222:ef7e::/48,2020:2222:efbc::/47,2121:2222:effc::/47,2121:2222:efe6::/47,10.136.52.248/32,10.82.219.222/32,10.248.96.52/32,10.248.96.51/32,10.80.125.220/32,10.102.146.94/32,10.102.135.134/32,10.245.252.3/32,10.167.120.90/32,10.103.158.138/32,10.196.90.224/32,10.181.40.227/32,10.214.148.202/32,10.213.167.130/32,10.211.171.56/32,10.152.17.50/32,10.39.67.131/32,1.78.202.38/32,10.104.157.86/32,10.154.62.120/32,100.227.14.57/32,100.193.202.110/32,100.193.202.107/32,100.229.129.47/32,100.215.241.6/32,100.3.73.34/32,100.124.189.106/32,100.107.8.22/32,100.166.222.235/32,100.54.227.161/32,100.171.234.22/32,100.219.217.52/32,100.167.191.129/32,100.81.194.190/32,100.142.237.81/32,100.101.113.49/32,100.243.43.86/32,100.243.43.82/32,100.108.98.72/32,100.108.98.66/32,100.108.98.65/32,100.108.98.116/32,10.136.52.244/31,10.82.223.206/31,10.82.219.220/31,10.248.96.54/31,10.248.96.48/31,10.102.143.184/31,10.102.141.2/31,10.238.168.144/31,10.211.65.184/31,10.211.65.180/31,10.211.171.58/31,10.211.171.48/31,1.71.142.114/31,100.144.236.24/31,100.115.101.242/31,100.4.164.28/31,100.171.234.20/31,100.243.43.88/31,100.243.43.84/31,100.243.43.80/31,100.108.98.120/31,100.108.98.118/31,10.82.219.216/30,10.248.96.40/30,10.102.143.188/30,10.30.90.128/30,10.77.183.108/30,10.196.65.52/30,10.211.65.176/30,10.211.171.52/30,100.115.97.216/30,100.115.96.84/30,100.115.96.200/30,100.115.101.48/30,100.84.209.128/30,100.84.209.124/30,100.84.209.12/30,100.4.164.24/30,100.171.234.24/30,100.171.234.16/30,100.219.217.48/30,100.108.98.68/30,100.108.98.112/30,100.241.99.112/30,100.34.180.76/30,10.82.219.208/29,10.102.143.176/29,10.238.168.152/29,10.59.192.112/29,10.48.115.136/29,10.17.120.200/29,10.107.207.144/29,10.120.145.144/29,10.122.217.120/29,10.103.142.192/29,10.196.91.240/29,10.134.2.168/29,10.127.51.184/29,10.127.32.192/29,10.154.62.112/29,100.144.236.16/29,100.186.1.128/29,100.118.100.224/29,100.84.210.8/29,100.4.164.16/29,10.136.51.0/28,10.136.37.128/28,10.200.61.240/28,10.112.218.0/28,10.102.145.112/28,10.102.143.160/28,10.238.168.128/28,10.211.65.160/28,10.211.171.32/28,10.154.8.224/28,10.154.62.96/28,10.154.57.0/28,100.144.236.0/28,100.115.103.16/28,100.229.128.144/28,100.4.164.0/28,100.171.234.0/28,100.219.217.32/28,100.243.43.64/28,100.108.98.96/28,100.216.221.64/28,100.241.99.96/28,10.138.3.64/27,10.136.52.64/27,10.136.37.160/27,10.248.96.0/27,10.213.163.128/27,10.195.4.96/27,10.196.90.192/27,10.154.54.192/27,10.154.54.160/27,10.123.32.224/27,100.144.236.32/27,100.214.8.224/27,100.178.9.96/27,10.136.52.128/26,10.136.52.0/26,10.248.96.64/26,10.30.243.0/26,10.195.63.128/26,10.123.32.128/26,100.192.1.0/26,100.173.250.64/26,100.108.98.192/26,10.136.37.0/25,10.248.96.128/25,10.238.169.128/25,10.196.91.0/25,100.192.1.128/25,10.137.166.0/24,10.136.53.0/24,10.136.36.0/24,10.248.110.0/24,10.238.183.0/24,10.238.170.0/24,10.46.168.0/24,10.30.242.0/24,10.195.65.0/24,10.228.35.0/24,100.99.254.0/24,100.14.212.0/24,100.46.19.0/24,100.174.4.0/24,100.0.0.0/24,100.125.135.0/24,100.243.250.0/24,100.108.105.0/24,100.104.191.0/24,10.137.138.0/23,10.136.54.0/23,10.136.38.0/23,10.136.224.0/23,10.248.108.0/23,1.3.34.0/23,10.30.240.0/23,10.72.118.0/23,10.6.34.0/23,10.196.70.0/23,100.82.118.0/23,100.67.66.0/23,100.83.248.0/23,100.83.216.0/23,100.104.188.0/23,10.137.124.0/22,10.30.196.0/22,10.147.116.0/22,10.180.144.0/22,10.142.236.0/22,10.195.88.0/22,10.209.164.0/22,100.110.236.0/22,100.89.120.0/22,100.86.4.0/22,100.93.196.0/22,100.10.188.0/22,10.139.248.0/21,10.138.232.0/21,10.136.200.0/21,10.136.0.0/21,10.147.120.0/21,10.228.176.0/21,100.0.16.0/21,10.147.96.0/20,10.228.160.0/20,100.145.48.0/20,100.131.48.0/20,100.126.224.0/20,10.137.32.0/19,10.6.192.0/19,100.168.0.0/16,100.18.0.0/15,100.16.0.0/12,100.128.0.0/10,100.64.0.0/10,172.0.0.0/8,fc00::/7]"
Updated by Victor Julien about 5 years ago
- Assignee set to Victor Julien
- Target version set to 70
Thanks Andy, I can reproduce the issue.
Updated by Victor Julien about 5 years ago
I did a small optimization in https://github.com/OISF/suricata/pull/3762, but this doesn't yet address the main issue.
Updated by Victor Julien over 1 year ago
- Target version changed from TBD to 8.0.0-beta1
Updated by Victor Julien 2 months ago
- Related to Bug #6376: Huge increase on Suricata load time with a lot of ip-only rules and bigger HOME_NET added
Updated by Victor Julien 2 months ago
@Andy Wick are you able to test out current git master? A fix has been merged for ticket #6376, which I suspect is the same issue.
Updated by Victor Julien 7 days ago
- Related to Optimization #6792: detect/port: port grouping is quite slow in worst cases added
Updated by Victor Julien 7 days ago
- Status changed from New to Closed
- Assignee deleted (
Victor Julien) - Target version deleted (
8.0.0-beta1)
Going to assume that this is resolved through #6792.
Actions