Project

General

Profile

Actions

Bug #4407

closed

threshold: slow startup on threshold.config with many addresses in suppression

Added by Victor Julien 6 months ago. Updated 5 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:
Needs backport to 5.0, Needs backport to 6.0

Description

A threshold config like

suppress gen_id 0, sig_id 0, track by_src, ip $SUPPRESS
suppress gen_id 1, sig_id 0, track by_src, ip $SUPPRESS

Combined with a large number of addresses in $SUPPRESS causes Suricata to load really slowly when using many rules.

  21,40%  Suricata-Main  suricata             [.] DetectAddressCmpIPv4
  20,53%  Suricata-Main  suricata             [.] DetectAddressCmp
  12,61%  Suricata-Main  suricata             [.] DetectAddressInsert

This appears to be caused by parsing the complex address string over and over again.

In per rule address parsing we cache. In threshold address parsing it appears this is not done, or is broken.

Steps to reproduce:

Create a config with many addresses:

#!/usr/bin/env python
# python cidr.py 192.168.1.1/24

import sys, struct, socket

(ip, cidr) = sys.argv[1].split('/')
cidr = int(cidr) 
host_bits = 32 - cidr
i = struct.unpack('>I', socket.inet_aton(ip))[0] # note the endianness
start = (i >> host_bits) << host_bits # clear the host bits
end = start | ((1 << host_bits) - 1)

header = "%YAML 1.1\n---\n" 
print(header)

final = "" 
cnt = 0 
i = 0 
s = "" 
# excludes the first and last address in the subnet
for r in range(start, end):
    x = socket.inet_ntoa(struct.pack('>I',r))
    if i == 0 or i % 100 == 0:
        if i != 0:
            s = s + "]\"" 
            final = final + "," + "$MYVAR" + str(cnt)
        else:
            final = "SUPPRESS: \"[$MYVAR" + str(cnt)
        print(s)
        s = "MYVAR" + str(cnt) + ": \"[" + x
        cnt += 1 
    else:
        s = s + "," + x 
    i += 1 
s = s + "]\"" 
final = final + "]\"" 
print(s)
print(final)

(taken from https://stackoverflow.com/a/44043448)

Run: python ipaddresses.py 1.0.0.0/22 > suppress.yaml

In your suricata.yaml add:

vars:
  # more specific is better for alert accuracy and performance
  address-groups:
    HOME_NET: "[192.168.0.0/16,10.0.0.0/8,172.16.0.0/12]" 
    #HOME_NET: "[192.168.0.0/16]" 
    #HOME_NET: "[10.0.0.0/8]" 
    #HOME_NET: "[172.16.0.0/12]" 
    #HOME_NET: "any" 

    EXTERNAL_NET: "!$HOME_NET" 
    #EXTERNAL_NET: "any" 

    HTTP_SERVERS: "$HOME_NET" 
    SMTP_SERVERS: "$HOME_NET" 
    SQL_SERVERS: "$HOME_NET" 
    DNS_SERVERS: "$HOME_NET" 
    TELNET_SERVERS: "$HOME_NET" 
    AIM_SERVERS: "$EXTERNAL_NET" 
    DC_SERVERS: "$HOME_NET" 
    DNP3_SERVER: "$HOME_NET" 
    DNP3_CLIENT: "$HOME_NET" 
    MODBUS_CLIENT: "$HOME_NET" 
    MODBUS_SERVER: "$HOME_NET" 
    ENIP_CLIENT: "$HOME_NET" 
    ENIP_SERVER: "$HOME_NET" 
    include: suppress.yaml

Finally load a larger ruleset, like ET/open.


Related issues

Copied to Bug #4414: threshold: slow startup on threshold.config with many addresses in suppressionClosedJeff LucovskyActions
Copied to Bug #4415: threshold: slow startup on threshold.config with many addresses in suppressionClosedShivani BhardwajActions
Actions #1

Updated by Victor Julien 6 months ago

  • Subject changed from threshold: slow startup threshold.config with many addresses in suppression to threshold: slow startup on threshold.config with many addresses in suppression
Actions #2

Updated by Jeff Lucovsky 6 months ago

  • Status changed from Assigned to In Progress
Actions #3

Updated by Jeff Lucovsky 6 months ago

  • Status changed from In Progress to In Review
Actions #4

Updated by Jeff Lucovsky 5 months ago

  • Copied to Bug #4414: threshold: slow startup on threshold.config with many addresses in suppression added
Actions #5

Updated by Jeff Lucovsky 5 months ago

  • Copied to Bug #4415: threshold: slow startup on threshold.config with many addresses in suppression added
Actions #6

Updated by Jeff Lucovsky 5 months ago

  • Status changed from In Review to Closed
Actions

Also available in: Atom PDF