Bug #4581
closedExcessive qsort/msort time when large number of rules using tls.fingerprint
Description
When a large number of rules using tls.fingerprint
are present, perf
shows Suricata spending most of its time in qsort
.
Is there room for an optimization that performs the sorting once to avoid sorting on the packet path?
Updated by Jeff Lucovsky about 3 years ago
This doesn't reproduce with modern toolchains (gcc
and libc
)
Updated by Victor Julien over 2 years ago
- Related to Support #3252: Matching a long list of tls.fingerprint fields is extremly CPU intensive added
Updated by Victor Julien over 2 years ago
Perhaps we can "fix" this by keeping the keyword but replace the actual implementation with the logic from tls.cert_fingerprint. We do something similar with the legacy uricontent
.
Updated by Jeff Lucovsky about 2 years ago
- Status changed from New to In Review
- Assignee set to Jeff Lucovsky
Updated by Jeff Lucovsky about 2 years ago
I'm using the rules at https://sslbl.abuse.ch/blacklist/sslblacklist.rules for comparing timing results.
Timing results with modifications
$ time src/suricata -c suricata.yaml -l /tmp/new -r ~/pcap/ -S ~/rules/blacklist.rules [1657190] 17/9/2022 -- 08:45:31 - (suricata.c:1146) <Notice> (LogVersion) -- This is Suricata version 7.0.0-dev running in USER mode [1657190] 17/9/2022 -- 08:45:32 - (tm-threads.c:1927) <Notice> (TmThreadWaitOnThreadInit) -- Threads created -> RX: 1 W: 16 FM: 1 FR: 1 Engine started. [1657190] 17/9/2022 -- 08:47:12 - (suricata.c:2774) <Notice> (SuricataMainLoop) -- Signal Received. Stopping engine. [1657204] 17/9/2022 -- 08:47:12 - (source-pcap-file.c:386) <Notice> (ReceivePcapFileThreadExitStats) -- Pcap-file module read 88 files, 24079025 packets, 4356793545 bytes real 1m41.540s user 11m12.750s sys 3m13.117s
Timing results from master commit bb2e11108b3ce5351f3abf45777c47fca37936af
$ time src/suricata -c suricata.yaml -l /tmp/orig -r ~/pcap/ -S ~/rules/blacklist.rules [1655193] 16/9/2022 -- 11:30:12 - (suricata.c:1146) <Notice> (LogVersion) -- This is Suricata version 7.0.0-dev (bb2e11108b 2022-08-30) running in USER mode [1655193] 16/9/2022 -- 11:30:12 - (tm-threads.c:1927) <Notice> (TmThreadWaitOnThreadInit) -- Threads created -> RX: 1 W: 16 FM: 1 FR: 1 Engine started. [1655193] 16/9/2022 -- 12:31:35 - (suricata.c:2774) <Notice> (SuricataMainLoop) -- Signal Received. Stopping engine. [1655207] 16/9/2022 -- 12:31:41 - (source-pcap-file.c:386) <Notice> (ReceivePcapFileThreadExitStats) -- Pcap-file module read 88 files, 24079025 packets, 4356793545 bytes real 61m29.628s user 894m56.266s sys 3m16.542s
Note: The following script was contributed by Justin Azoff (justin@corelight.com) to create a rule file using tls.fingerprint
. The script can be used to generate a file with an arbitrary number of rules.
#!/usr/bin/env python3 import sys import random try: num = int(sys.argv[1]) except: num = 200 try: comment_extra = int(sys.argv[2]) except: comment_extra = 0 def random_fingerprint(): return ":".join([f"{x:02x}" for x in open("/dev/urandom",'rb').read(20)]) template='''alert tls $EXTERNAL_NET any -> $HOME_NET any (msg:"Test rule {rule}"; flow: established; tls.fingerprint:"{fingerprint}"; sid:20{rule:05d}; rev:1;)''' for rule in range(num): print(template.format(rule=rule, fingerprint=random_fingerprint()))
Updated by Jeff Lucovsky about 2 years ago
- Status changed from In Review to Closed
Updated by Jeff Lucovsky about 2 years ago
- Status changed from Closed to Resolved
Updated by Victor Julien about 2 years ago
- Target version set to 7.0.0-beta1
Updated by Shivani Bhardwaj about 2 years ago
- Label deleted (
Needs backport to 6.0)
Updated by Victor Julien almost 2 years ago
- Related to Bug #5565: Excessive qsort/msort time when large number of rules using tls.fingerprint (6.0.x backport) added
Updated by Victor Julien almost 2 years ago
- Status changed from Resolved to Closed
Updated by Jeff Lucovsky over 1 year ago
- Status changed from Closed to Resolved
- Label Needs backport to 6.0 added
Re-opening for 6.0.x backport
Updated by Victor Julien over 1 year ago
- Status changed from Resolved to Closed