Project

General

Profile

Actions

Bug #4407

closed

threshold: slow startup on threshold.config with many addresses in suppression

Added by Victor Julien about 3 years ago. Updated almost 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:
Needs backport to 5.0, Needs backport to 6.0

Description

A threshold config like

suppress gen_id 0, sig_id 0, track by_src, ip $SUPPRESS
suppress gen_id 1, sig_id 0, track by_src, ip $SUPPRESS

Combined with a large number of addresses in $SUPPRESS causes Suricata to load really slowly when using many rules.

  21,40%  Suricata-Main  suricata             [.] DetectAddressCmpIPv4
  20,53%  Suricata-Main  suricata             [.] DetectAddressCmp
  12,61%  Suricata-Main  suricata             [.] DetectAddressInsert

This appears to be caused by parsing the complex address string over and over again.

In per rule address parsing we cache. In threshold address parsing it appears this is not done, or is broken.

Steps to reproduce:

Create a config with many addresses:

#!/usr/bin/env python
# python cidr.py 192.168.1.1/24

import sys, struct, socket

(ip, cidr) = sys.argv[1].split('/')
cidr = int(cidr) 
host_bits = 32 - cidr
i = struct.unpack('>I', socket.inet_aton(ip))[0] # note the endianness
start = (i >> host_bits) << host_bits # clear the host bits
end = start | ((1 << host_bits) - 1)

header = "%YAML 1.1\n---\n" 
print(header)

final = "" 
cnt = 0 
i = 0 
s = "" 
# excludes the first and last address in the subnet
for r in range(start, end):
    x = socket.inet_ntoa(struct.pack('>I',r))
    if i == 0 or i % 100 == 0:
        if i != 0:
            s = s + "]\"" 
            final = final + "," + "$MYVAR" + str(cnt)
        else:
            final = "SUPPRESS: \"[$MYVAR" + str(cnt)
        print(s)
        s = "MYVAR" + str(cnt) + ": \"[" + x
        cnt += 1 
    else:
        s = s + "," + x 
    i += 1 
s = s + "]\"" 
final = final + "]\"" 
print(s)
print(final)

(taken from https://stackoverflow.com/a/44043448)

Run: python ipaddresses.py 1.0.0.0/22 > suppress.yaml

In your suricata.yaml add:

vars:
  # more specific is better for alert accuracy and performance
  address-groups:
    HOME_NET: "[192.168.0.0/16,10.0.0.0/8,172.16.0.0/12]" 
    #HOME_NET: "[192.168.0.0/16]" 
    #HOME_NET: "[10.0.0.0/8]" 
    #HOME_NET: "[172.16.0.0/12]" 
    #HOME_NET: "any" 

    EXTERNAL_NET: "!$HOME_NET" 
    #EXTERNAL_NET: "any" 

    HTTP_SERVERS: "$HOME_NET" 
    SMTP_SERVERS: "$HOME_NET" 
    SQL_SERVERS: "$HOME_NET" 
    DNS_SERVERS: "$HOME_NET" 
    TELNET_SERVERS: "$HOME_NET" 
    AIM_SERVERS: "$EXTERNAL_NET" 
    DC_SERVERS: "$HOME_NET" 
    DNP3_SERVER: "$HOME_NET" 
    DNP3_CLIENT: "$HOME_NET" 
    MODBUS_CLIENT: "$HOME_NET" 
    MODBUS_SERVER: "$HOME_NET" 
    ENIP_CLIENT: "$HOME_NET" 
    ENIP_SERVER: "$HOME_NET" 
    include: suppress.yaml

Finally load a larger ruleset, like ET/open.


Related issues 2 (0 open2 closed)

Copied to Suricata - Bug #4414: threshold: slow startup on threshold.config with many addresses in suppressionClosedJeff LucovskyActions
Copied to Suricata - Bug #4415: threshold: slow startup on threshold.config with many addresses in suppressionClosedShivani BhardwajActions
Actions

Also available in: Atom PDF