Suricata

The engine takes an excessive amount of time to load when I try to load all VRT and emerging rules, with the default suricata.yaml file. For example on the line below, the pcap that was processed was only 3k and yet it took the engine 952 seconds to analyze the file.

ulimit -c unlimited; src/suricata -c suricata.yaml -r ./WatchFire_Appscan_7.0_ActiveX_Multiple_Insecure_Methods_Exploit.pcap-fuzz-2010-02-08-04-10-35 -l ./ -s /home/coz/downloads/current-all-blah.rules,952,0

Running the command manually it seems that engine hangs during sig grouping. Bt seems to show the same thing on another run after 6 minutes we are still sorting signatures. I realize that 14k signatures is a lot but this is excessive.

GNU gdb (GDB) 7.0-ubuntu
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/&gt;...
attach: No such file or directory.
Attaching to process 20350
Reading symbols from /home/coz/downloads/suricatafuzz1/src/suricata...done.
Reading symbols from /usr/local/lib/libhtp-0.2.so.1...done.
Loaded symbols for /usr/local/lib/libhtp-0.2.so.1
Reading symbols from /usr/lib/libpcap.so.0.8...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libpcap.so.0.8
Reading symbols from /usr/local/lib/libpfring.so...done.
Loaded symbols for /usr/local/lib/libpfring.so
Reading symbols from /usr/lib/libnet.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libnet.so.1
Reading symbols from /lib/libpthread.so.0...Reading symbols from /usr/lib/debug/lib/libpthread-2.10.1.so...done.
[Thread debugging using libthread_db enabled]
(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /usr/lib/libyaml-0.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libyaml-0.so.1
Reading symbols from /lib/libpcre.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib/libpcre.so.3
Reading symbols from /lib/libc.so.6...Reading symbols from /usr/lib/debug/lib/libc-2.10.1.so...done.
(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/lib/ld-2.10.1.so...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
DetectAddressCmpIPv4 (a=0xca54c30, b=0x2be4fd0) at detect-engine-address-ipv4.c:51
51 } else if (a_ip1 <= b_ip1 && a_ip2 >= b_ip2) {
(gdb) bt full
#0 DetectAddressCmpIPv4 (a=0xca54c30, b=0x2be4fd0) at detect-engine-address-ipv4.c:51
a_ip1 = 3221226240
a_ip2 = 3323068415
b_ip1 = 1208659156
b_ip2 = 1208659156
#1 0x000000000042b0f2 in DetectAddressCmp (a=0xca54c30, b=0x2be4fd0) at detect-engine-address.c:1395
No locals.
#2 0x0000000000428942 in DetectAddressLookupInList (head=0x9e5fa60, gr=0x2be4fd0) at detect-engine-address.c:216
cur = 0xca54c30
#3 0x0000000000421245 in BuildDestinationAddressHeads (de_ctx=0x282c9e0, head=0xb5ef110, family=2, dsize=0, flow=1) at detect.c:1873
tmp_gr_list = 0x9e5fa60
sig = 961
groups = 2
tmp_s = 0x2be23b0
gr = 0xc219ed0
sgr = 0x2be4fd0
lookup_gr = 0x0
max_idx = 961
grhead = 0xc219f80
grdsthead = 0xbf702f0
grsighead = 0x2be4910
#4 0x0000000000422dd5 in SigAddressPrepareStage3 (de_ctx=0x282c9e0) at detect.c:2390
r = 0
FUNCTION = "SigAddressPrepareStage3"
ds = 0
f = 1
proto = 132
#5 0x0000000000424c14 in SigGroupBuild (de_ctx=0x282c9e0) at detect.c:2815
No locals.
#6 0x000000000041d149 in SigLoadSignatures (de_ctx=0x282c9e0, sig_file=0x7fff2522d615 "/home/coz/downloads/current-all-blah.rules") at detect.c:412
prevsig = 0x2a702c0
sig = 0x2a70870
rule_files = 0x2142250
file = 0x0
ret = 0
r = 14362
cnt = 14362
cntf = 71
sigtotal = 14516
sfile = 0x2a70d60 ""
FUNCTION = "SigLoadSignatures"
#7 0x0000000000404f15 in main (argc=9, argv=0x7fff2522ce28) at suricata.c:644
opt = 1
pcap_file = 0x7fff2522d5d6 "./fuzz-2008-02-22-13184.pcap-fuzz-2010-02-08-16-22-44"
pcap_dev = 0x0
pfring_dev = 0x0
sig_file = 0x7fff2522d615 "/home/coz/downloads/current-all-blah.rules"
nfq_id = 0
conf_filename = 0x7fff2522d5c5 "suricata.yaml"
dump_config = 0
list_unittests = 0
daemon = 0
log_dir = 0x212e1a0 "./"
buf = {st_dev = 2055, st_ino = 5530778, st_nlink = 7, st_mode = 16877, st_uid = 1000, st_gid = 1000, pad0 = 0, st_rdev = 0, st_size = 4096, st_blksize = 4096, st_blocks = 8, st_atim = {tv_sec = 1265667764, tv_nsec = 0},
st_mtim = {tv_sec = 1265667764, tv_nsec = 0}, st_ctim = {tv_sec = 1265667764, tv_nsec = 0}, __unused = {0, 0, 0}}
long_opts = {{name = 0x4b2988 "dump-config", has_arg = 0, flag = 0x7fff2522c900, val = 1}, {name = 0x4b2994 "pfring-int", has_arg = 1, flag = 0x0, val = 0}, {name = 0x4b299f "pfring-clusterid", has_arg = 1, flag = 0x0,
---Type <return> to continue, or q <return> to quit--
val = 0}, {name = 0x4b29b0 "unittest-filter", has_arg = 1, flag = 0x0, val = 85}, {name = 0x4b29c0 "list-unittests", has_arg = 0, flag = 0x7fff2522c8fc, val = 1}, {name = 0x4b29cf "init-errors-fatal", has_arg = 0,
flag = 0x0, val = 0}, {name = 0x4b29e1 "fatal-unittests", has_arg = 0, flag = 0x0, val = 0}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}}
option_index = 0
short_opts = "c:Dhi:l:q:r:us:U:V"
__FUNCTION = "main"
c = 255 '\377'
i = 50
de_ctx = 0x282c9e0
start_time = {tv_sec = -481287187988480, tv_usec = 139985586884608}

I'm creating the rules file containing all VRT and emerging rules with following script.

#!/bin/sh
oinkmaster.pl -C /etc/oinkmaster-vrt.conf -o /etc/suricata/
oinkmaster.pl -C /etc/oinkmaster-emerging.conf -o /etc/suricata/
create-sidmap.pl /etc/suricata/> /testscripts/ubersid-msg.map

list=`ls /etc/suricata/*.rules`
for file in $list
do
cat $file >> /testscripts/current-all-blah.rules
done