Bug #4581
closedExcessive qsort/msort time when large number of rules using tls.fingerprint
Description
When a large number of rules using tls.fingerprint are present, perf shows Suricata spending most of its time in qsort.
Is there room for an optimization that performs the sorting once to avoid sorting on the packet path?
Updated by Jeff Lucovsky about 4 years ago
This doesn't reproduce with modern toolchains (gcc and libc)
Updated by Victor Julien about 3 years ago
- Related to Support #3252: Matching a long list of tls.fingerprint fields is extremly CPU intensive added
Updated by Victor Julien about 3 years ago
Perhaps we can "fix" this by keeping the keyword but replace the actual implementation with the logic from tls.cert_fingerprint. We do something similar with the legacy uricontent.
Updated by Jeff Lucovsky about 3 years ago
- Status changed from New to In Review
- Assignee set to Jeff Lucovsky
Updated by Jeff Lucovsky about 3 years ago
I'm using the rules at https://sslbl.abuse.ch/blacklist/sslblacklist.rules for comparing timing results.
Timing results with modifications
$ time src/suricata -c suricata.yaml -l /tmp/new -r ~/pcap/ -S ~/rules/blacklist.rules [1657190] 17/9/2022 -- 08:45:31 - (suricata.c:1146) <Notice> (LogVersion) -- This is Suricata version 7.0.0-dev running in USER mode [1657190] 17/9/2022 -- 08:45:32 - (tm-threads.c:1927) <Notice> (TmThreadWaitOnThreadInit) -- Threads created -> RX: 1 W: 16 FM: 1 FR: 1 Engine started. [1657190] 17/9/2022 -- 08:47:12 - (suricata.c:2774) <Notice> (SuricataMainLoop) -- Signal Received. Stopping engine. [1657204] 17/9/2022 -- 08:47:12 - (source-pcap-file.c:386) <Notice> (ReceivePcapFileThreadExitStats) -- Pcap-file module read 88 files, 24079025 packets, 4356793545 bytes real 1m41.540s user 11m12.750s sys 3m13.117s
Timing results from master commit bb2e11108b3ce5351f3abf45777c47fca37936af
$ time src/suricata -c suricata.yaml -l /tmp/orig -r ~/pcap/ -S ~/rules/blacklist.rules [1655193] 16/9/2022 -- 11:30:12 - (suricata.c:1146) <Notice> (LogVersion) -- This is Suricata version 7.0.0-dev (bb2e11108b 2022-08-30) running in USER mode [1655193] 16/9/2022 -- 11:30:12 - (tm-threads.c:1927) <Notice> (TmThreadWaitOnThreadInit) -- Threads created -> RX: 1 W: 16 FM: 1 FR: 1 Engine started. [1655193] 16/9/2022 -- 12:31:35 - (suricata.c:2774) <Notice> (SuricataMainLoop) -- Signal Received. Stopping engine. [1655207] 16/9/2022 -- 12:31:41 - (source-pcap-file.c:386) <Notice> (ReceivePcapFileThreadExitStats) -- Pcap-file module read 88 files, 24079025 packets, 4356793545 bytes real 61m29.628s user 894m56.266s sys 3m16.542s
Note: The following script was contributed by Justin Azoff (justin@corelight.com) to create a rule file using tls.fingerprint. The script can be used to generate a file with an arbitrary number of rules.
#!/usr/bin/env python3
import sys
import random
try:
num = int(sys.argv[1])
except:
num = 200
try:
comment_extra = int(sys.argv[2])
except:
comment_extra = 0
def random_fingerprint():
return ":".join([f"{x:02x}" for x in open("/dev/urandom",'rb').read(20)])
template='''alert tls $EXTERNAL_NET any -> $HOME_NET any (msg:"Test rule {rule}"; flow: established; tls.fingerprint:"{fingerprint}"; sid:20{rule:05d}; rev:1;)'''
for rule in range(num):
print(template.format(rule=rule, fingerprint=random_fingerprint()))
Updated by Jeff Lucovsky about 3 years ago
- Status changed from In Review to Closed
Updated by Jeff Lucovsky about 3 years ago
- Status changed from Closed to Resolved
Updated by Victor Julien about 3 years ago
- Target version set to 7.0.0-beta1
Updated by Shivani Bhardwaj about 3 years ago
- Label deleted (
Needs backport to 6.0)
Updated by Victor Julien almost 3 years ago
- Related to Bug #5565: Excessive qsort/msort time when large number of rules using tls.fingerprint (6.0.x backport) added
Updated by Victor Julien almost 3 years ago
- Status changed from Resolved to Closed
Updated by Jeff Lucovsky over 2 years ago
- Status changed from Closed to Resolved
- Label Needs backport to 6.0 added
Re-opening for 6.0.x backport
Updated by Victor Julien over 2 years ago
- Status changed from Resolved to Closed