Project

General

Profile

Bug #4414

threshold: slow startup on threshold.config with many addresses in suppression

Added by Jeff Lucovsky about 1 month ago. Updated 13 days ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

A threshold config like

suppress gen_id 0, sig_id 0, track by_src, ip $SUPPRESS
suppress gen_id 1, sig_id 0, track by_src, ip $SUPPRESS

Combined with a large number of addresses in $SUPPRESS causes Suricata to load really slowly when using many rules.

  21,40%  Suricata-Main  suricata             [.] DetectAddressCmpIPv4
  20,53%  Suricata-Main  suricata             [.] DetectAddressCmp
  12,61%  Suricata-Main  suricata             [.] DetectAddressInsert

This appears to be caused by parsing the complex address string over and over again.

In per rule address parsing we cache. In threshold address parsing it appears this is not done, or is broken.

Steps to reproduce:

Create a config with many addresses:

#!/usr/bin/env python
# python cidr.py 192.168.1.1/24

import sys, struct, socket

(ip, cidr) = sys.argv[1].split('/')
cidr = int(cidr) 
host_bits = 32 - cidr
i = struct.unpack('>I', socket.inet_aton(ip))[0] # note the endianness
start = (i >> host_bits) << host_bits # clear the host bits
end = start | ((1 << host_bits) - 1)

header = "%YAML 1.1\n---\n" 
print(header)

final = "" 
cnt = 0 
i = 0 
s = "" 
# excludes the first and last address in the subnet
for r in range(start, end):
    x = socket.inet_ntoa(struct.pack('>I',r))
    if i == 0 or i % 100 == 0:
        if i != 0:
            s = s + "]\"" 
            final = final + "," + "$MYVAR" + str(cnt)
        else:
            final = "SUPPRESS: \"[$MYVAR" + str(cnt)
        print(s)
        s = "MYVAR" + str(cnt) + ": \"[" + x
        cnt += 1 
    else:
        s = s + "," + x 
    i += 1 
s = s + "]\"" 
final = final + "]\"" 
print(s)
print(final)

(taken from https://stackoverflow.com/a/44043448)

Run: python ipaddresses.py 1.0.0.0/22 > suppress.yaml

In your suricata.yaml add:

vars:
  # more specific is better for alert accuracy and performance
  address-groups:
    HOME_NET: "[192.168.0.0/16,10.0.0.0/8,172.16.0.0/12]" 
    #HOME_NET: "[192.168.0.0/16]" 
    #HOME_NET: "[10.0.0.0/8]" 
    #HOME_NET: "[172.16.0.0/12]" 
    #HOME_NET: "any" 

    EXTERNAL_NET: "!$HOME_NET" 
    #EXTERNAL_NET: "any" 

    HTTP_SERVERS: "$HOME_NET" 
    SMTP_SERVERS: "$HOME_NET" 
    SQL_SERVERS: "$HOME_NET" 
    DNS_SERVERS: "$HOME_NET" 
    TELNET_SERVERS: "$HOME_NET" 
    AIM_SERVERS: "$EXTERNAL_NET" 
    DC_SERVERS: "$HOME_NET" 
    DNP3_SERVER: "$HOME_NET" 
    DNP3_CLIENT: "$HOME_NET" 
    MODBUS_CLIENT: "$HOME_NET" 
    MODBUS_SERVER: "$HOME_NET" 
    ENIP_CLIENT: "$HOME_NET" 
    ENIP_SERVER: "$HOME_NET" 
    include: suppress.yaml

Finally load a larger ruleset, like ET/open.


Related issues

Copied from Bug #4407: threshold: slow startup on threshold.config with many addresses in suppressionClosedJeff LucovskyActions
#1

Updated by Jeff Lucovsky about 1 month ago

  • Copied from Bug #4407: threshold: slow startup on threshold.config with many addresses in suppression added
#2

Updated by Victor Julien 20 days ago

  • Target version changed from 6.0.3 to 5.0.7
#3

Updated by Jeff Lucovsky 17 days ago

  • Status changed from Assigned to In Progress

Cherry-pick: 11f9cc6, e873632, 02ceac8, 2893b04

#4

Updated by Jeff Lucovsky 15 days ago

  • Status changed from In Progress to In Review
#5

Updated by Jeff Lucovsky 13 days ago

  • Status changed from In Review to Closed

Also available in: Atom PDF