Project

General

Profile

Actions

Bug #2908

open

ip only rules cause suricata to take 17 minutes to start

Added by Andy Wick over 3 years ago. Updated 5 days ago.

Status:
New
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

We are trying to run with 300+ CIDRs in our home net with about 50k rules (et pro and others). Suricata 4.1.2 & 4.1.3 takes about 17 minutes between when we get the rules loaded and when we get the af packet threads are now listening print out. Using gperftools it says that over 97% of the time is spent in IPOnlyCIDRItemInsertReal. Are we doing something wrong?

A quick glance at the code makes it looks like a link list insertion sort is being used to sort these 300+ cidrs for every single rule?

Seems like some possible solutions
  • We could presort our config from max netmask to smallest, so the sort would be O(n). We have our ipv6 ips last in the list, so I bet currently we are worst case near O(n^2)
  • The code could switch to a qsort instead of link list insertion sort
  • The code could cache the list

Any help would be great!

Actions #1

Updated by Victor Julien over 3 years ago

So this is a normal ruleset like ET used with a large HOME_NET? Are you able to share your HOME_NET setting, or create a fake one that has the same effect?

Actions #2

Updated by Andy Wick over 3 years ago

To make it easy to repo I switched to just the full et pro rule set and nothing else. Which outputs

45259 signatures processed. 1225 are IP-only rules, 17090 are inspecting packet payload, 31407 inspect application layer, 0 are decoder event only

I changed LoadSignatures to do a exit instead of returning for all times below.

So running with 
time /nids/suricata/bin/suricata -c /nids/suricata/etc/suricata.yaml --af-packet -v

Our original ip list:
real    29m15.512s
user    28m49.707s
sys     0m25.756s

Then I sorted the list with: sort -t/ -k2 -n -r
real    11m20.801s
user    11m18.115s
sys     0m2.682s

So sorting helped a lot. I tried to make a sorted fake list, which isn't quite as slow, but still almost 8m.

real    7m58.549s
user    7m56.561s
sys     0m1.986s

"[1111:2222:3333:7fff::a:1/128,1111:2222:3333:7eff::a:1/128,1212:2222:3333:7fff::a:1/128,1212:2222:3333:7eff::a:1/128,1313:2222:3333:ff::a:2/128,1313:2222:3333:ff::a:1/128,1313:2222:3333:7cff::a:1/128,1111:2222:3333:7ffe::/127,1111:2222:3333:7ffc::/127,1212:2222:3333:7ffe::/127,1212:2222:3333:7ffc::/127,1414:2222:f024:1ff::/64,1111:2222:3333::/64,1111:2222:3333:7f02::/64,1414:2222:efca:7ffc::/64,1414:2222:efca:7f02::/64,1414:2222:efca:7eff::/64,1414:2222:efca:1::/64,1212:2222:3333:7f02::/64,1212:2222:3333:2::/64,1212:2222:3333:1::/64,1414:2222:ef98:7ffd::/64,1414:2222:ef98:7eff::/64,1414:2222:ef98:7bff::/64,1414:2222:ef98:7808::/64,1414:2222:ef98:7801::/64,1414:2222:ef98:110::/64,1414:2222:ef98:10c::/64,1515:2222:3333:8::/64,1515:2222:3333:7ffd::/64,1515:2222:3333:7eff::/64,1515:2222:3333:7dff::/64,1515:2222:3333:7d00::/64,1616:2222:3333:7ffd::/64,1616:2222:3333:7eff::/64,1616:2222:3333:7dff::/64,1616:2222:3333:7d00::/64,1616:2222:3333:7801::/64,1717:2222:3333:7eff::/64,1717:2222:3333:7dff::/64,1717:2222:3333:7d00::/64,1717:2222:3333:7806::/64,1717:2222:3333:7801::/64,1717:2222:3333:1fc::/64,1717:2222:3333:14::/64,1313:2222:3333:fe::/64,1313:2222:3333:8::/64,1313:2222:3333:7ffd::/64,1313:2222:3333:7eff::/64,1313:2222:3333:7dff::/64,1313:2222:3333:7d00::/64,1313:2222:3333:7cfe::/64,1313:2222:3333:7cfd::/64,1313:2222:3333:7800::/64,1313:2222:3333:11a::/64,1313:2222:3333:119::/64,1818:2222:3333:1007::/64,1919:2222:3333:a::/64,1111:2222:3333:7f00::/63,1414:2222:efca:7ffe::/63,1414:2222:efca:7f00::/63,1414:2222:efca:2::/63,1212:2222:3333:7f00::/63,1414:2222:ef98:7ffe::/63,1414:2222:ef98:7f00::/63,1414:2222:ef98:78fe::/63,1414:2222:ef98:7802::/63,1515:2222:3333:7ffe::/63,1515:2222:3333:7f00::/63,1616:2222:3333:7ffe::/63,1616:2222:3333:78fe::/63,1616:2222:3333:7802::/63,1717:2222:3333:7ffe::/63,1717:2222:3333:7f00::/63,1717:2222:3333:7804::/63,1717:2222:3333:7802::/63,1717:2222:3333:1fe::/63,1717:2222:3333:1f4::/63,1313:2222:3333:fc::/63,1313:2222:3333:a::/63,1313:2222:3333:7ffe::/63,1313:2222:3333:7f00::/63,1313:2222:3333:7c08::/63,1313:2222:3333:1fe::/63,1414:2222:ef98:fc::/62,1414:2222:ef98:7cfc::/62,1414:2222:ef98:7c00::/62,1414:2222:ef98:7b00::/62,1414:2222:ef98:7804::/62,1414:2222:ef98:1fc::/62,1414:2222:ef98:108::/62,1515:2222:3333:fc::/62,1616:2222:3333:fc::/62,1616:2222:3333:7f00::/62,1616:2222:3333:18::/62,1717:2222:3333:1f0::/62,1313:2222:3333:7804::/62,1313:2222:3333:110::/62,1414:2222:ef98::/61,1414:2222:ef98:100::/61,1515:2222:3333::/61,1616:2222:3333:10::/61,1717:2222:3333:100::/61,1313:2222:3333::/61,1313:2222:3333:7c00::/61,1313:2222:3333:7808::/61,1616:2222:3333::/60,1313:2222:3333:7810::/60,1313:2222:3333:100::/60,1313:2222:3333:7820::/59,1313:2222:3333:7840::/58,1313:2222:3333:7880::/57,1414:2222:ef98:7d00::/56,1313:2222:3333:7900::/56,1313:2222:3333:7a00::/55,1515:2222:3333:8000::/49,1616:2222:3333:8000::/49,1414:2222:efc4::/48,1414:2222:efc2::/48,1414:2222:ef84::/48,2020:2222:efba::/48,2020:2222:efb8::/48,2020:2222:efaf::/48,2020:2222:efa8::/48,2020:2222:efa7::/48,2020:2222:efa5::/48,2020:2222:ef96::/48,2121:2222:fbff::/48,2121:2222:eff5::/48,2121:2222:efeb::/48,2121:2222:efd8::/48,2121:2222:efad::/48,2121:2222:ef99::/48,2121:2222:ef94::/48,2121:2222:ef8f::/48,2121:2222:ef83::/48,2121:2222:ef7e::/48,2020:2222:efbc::/47,2121:2222:effc::/47,2121:2222:efe6::/47,10.136.52.248/32,10.82.219.222/32,10.248.96.52/32,10.248.96.51/32,10.80.125.220/32,10.102.146.94/32,10.102.135.134/32,10.245.252.3/32,10.167.120.90/32,10.103.158.138/32,10.196.90.224/32,10.181.40.227/32,10.214.148.202/32,10.213.167.130/32,10.211.171.56/32,10.152.17.50/32,10.39.67.131/32,1.78.202.38/32,10.104.157.86/32,10.154.62.120/32,100.227.14.57/32,100.193.202.110/32,100.193.202.107/32,100.229.129.47/32,100.215.241.6/32,100.3.73.34/32,100.124.189.106/32,100.107.8.22/32,100.166.222.235/32,100.54.227.161/32,100.171.234.22/32,100.219.217.52/32,100.167.191.129/32,100.81.194.190/32,100.142.237.81/32,100.101.113.49/32,100.243.43.86/32,100.243.43.82/32,100.108.98.72/32,100.108.98.66/32,100.108.98.65/32,100.108.98.116/32,10.136.52.244/31,10.82.223.206/31,10.82.219.220/31,10.248.96.54/31,10.248.96.48/31,10.102.143.184/31,10.102.141.2/31,10.238.168.144/31,10.211.65.184/31,10.211.65.180/31,10.211.171.58/31,10.211.171.48/31,1.71.142.114/31,100.144.236.24/31,100.115.101.242/31,100.4.164.28/31,100.171.234.20/31,100.243.43.88/31,100.243.43.84/31,100.243.43.80/31,100.108.98.120/31,100.108.98.118/31,10.82.219.216/30,10.248.96.40/30,10.102.143.188/30,10.30.90.128/30,10.77.183.108/30,10.196.65.52/30,10.211.65.176/30,10.211.171.52/30,100.115.97.216/30,100.115.96.84/30,100.115.96.200/30,100.115.101.48/30,100.84.209.128/30,100.84.209.124/30,100.84.209.12/30,100.4.164.24/30,100.171.234.24/30,100.171.234.16/30,100.219.217.48/30,100.108.98.68/30,100.108.98.112/30,100.241.99.112/30,100.34.180.76/30,10.82.219.208/29,10.102.143.176/29,10.238.168.152/29,10.59.192.112/29,10.48.115.136/29,10.17.120.200/29,10.107.207.144/29,10.120.145.144/29,10.122.217.120/29,10.103.142.192/29,10.196.91.240/29,10.134.2.168/29,10.127.51.184/29,10.127.32.192/29,10.154.62.112/29,100.144.236.16/29,100.186.1.128/29,100.118.100.224/29,100.84.210.8/29,100.4.164.16/29,10.136.51.0/28,10.136.37.128/28,10.200.61.240/28,10.112.218.0/28,10.102.145.112/28,10.102.143.160/28,10.238.168.128/28,10.211.65.160/28,10.211.171.32/28,10.154.8.224/28,10.154.62.96/28,10.154.57.0/28,100.144.236.0/28,100.115.103.16/28,100.229.128.144/28,100.4.164.0/28,100.171.234.0/28,100.219.217.32/28,100.243.43.64/28,100.108.98.96/28,100.216.221.64/28,100.241.99.96/28,10.138.3.64/27,10.136.52.64/27,10.136.37.160/27,10.248.96.0/27,10.213.163.128/27,10.195.4.96/27,10.196.90.192/27,10.154.54.192/27,10.154.54.160/27,10.123.32.224/27,100.144.236.32/27,100.214.8.224/27,100.178.9.96/27,10.136.52.128/26,10.136.52.0/26,10.248.96.64/26,10.30.243.0/26,10.195.63.128/26,10.123.32.128/26,100.192.1.0/26,100.173.250.64/26,100.108.98.192/26,10.136.37.0/25,10.248.96.128/25,10.238.169.128/25,10.196.91.0/25,100.192.1.128/25,10.137.166.0/24,10.136.53.0/24,10.136.36.0/24,10.248.110.0/24,10.238.183.0/24,10.238.170.0/24,10.46.168.0/24,10.30.242.0/24,10.195.65.0/24,10.228.35.0/24,100.99.254.0/24,100.14.212.0/24,100.46.19.0/24,100.174.4.0/24,100.0.0.0/24,100.125.135.0/24,100.243.250.0/24,100.108.105.0/24,100.104.191.0/24,10.137.138.0/23,10.136.54.0/23,10.136.38.0/23,10.136.224.0/23,10.248.108.0/23,1.3.34.0/23,10.30.240.0/23,10.72.118.0/23,10.6.34.0/23,10.196.70.0/23,100.82.118.0/23,100.67.66.0/23,100.83.248.0/23,100.83.216.0/23,100.104.188.0/23,10.137.124.0/22,10.30.196.0/22,10.147.116.0/22,10.180.144.0/22,10.142.236.0/22,10.195.88.0/22,10.209.164.0/22,100.110.236.0/22,100.89.120.0/22,100.86.4.0/22,100.93.196.0/22,100.10.188.0/22,10.139.248.0/21,10.138.232.0/21,10.136.200.0/21,10.136.0.0/21,10.147.120.0/21,10.228.176.0/21,100.0.16.0/21,10.147.96.0/20,10.228.160.0/20,100.145.48.0/20,100.131.48.0/20,100.126.224.0/20,10.137.32.0/19,10.6.192.0/19,100.168.0.0/16,100.18.0.0/15,100.16.0.0/12,100.128.0.0/10,100.64.0.0/10,172.0.0.0/8,fc00::/7]" 

Actions #3

Updated by Victor Julien over 3 years ago

  • Assignee set to Victor Julien
  • Target version set to 70

Thanks Andy, I can reproduce the issue.

Actions #4

Updated by Victor Julien over 3 years ago

I did a small optimization in https://github.com/OISF/suricata/pull/3762, but this doesn't yet address the main issue.

Actions #5

Updated by Victor Julien about 2 years ago

  • Target version changed from 70 to TBD
Actions #6

Updated by Victor Julien 5 days ago

  • Target version changed from TBD to 8.0beta1
Actions

Also available in: Atom PDF