Project

General

Profile

Actions

Bug #4567

open

crashed when high traffic in hyperscan:doNormal16 (mode=CALLBACK_OUTPUT, do_accel=0 '\000', s=1, #326

Added by eric fool about 2 months ago. Updated about 2 months ago.

Status:
New
Priority:
High
Assignee:
-
Target version:
-
Affected Versions:
Effort:
Difficulty:
Label:

Description

The hs was used in suricata of IPS worker mode, when the http traffic up to 2Gbps it crashed with only one thread,or less traffic with 4 threads。the core dump is follow:

Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `./suricata --runmode workers Q 4 -c suricata.yaml --set mpm-algo=hs'.
Program terminated with signal 11, Segmentation fault.
#0 doNormal16 (mode=CALLBACK_OUTPUT, do_accel=0 '\000', s=1,
end=0x7fa59872864e <Address 0x7fa59872864e out of bounds>, c_inout=, m=0x126eb600)
at /root/hyperscan/src/nfa/mcclellan.c:138
138 u8 cprime = m
>remap[*c];
Missing separate debuginfos, use: debuginfo-install file-libs-5.11-31.el7.x86_64 glib2-2.42.2-5.el7.x86_64 glibc-2.17-196.tl2.3.x86_64 gmime-2.6.23-1.el7.x86_64 gpgme-1.3.2-5.el7.x86_64 libassuan-2.1.0-3.el7.x86_64 libcap-ng-0.7.5-4.el7.x86_64 libffi-3.0.13-16.el7.x86_64 libgcc-4.8.5-39.tl2.1.x86_64 libgpg-error-1.12-3.el7.x86_64 libpcap-1.5.3-8.el7.x86_64 libselinux-2.2.2-6.el7.x86_64 libstdc++-4.8.5-39.tl2.1.x86_64 libyaml-0.1.4-11.el7_0.x86_64 luajit-2.0.4-3.el7.x86_64 lz4-1.7.5-2.tl2.x86_64 nspr-4.10.8-2.el7_1.x86_64 nss-3.19.1-19.el7_2.x86_64 nss-softokn-3.16.2.3-13.el7_1.x86_64 nss-softokn-freebl-3.16.2.3-13.el7_1.x86_64 nss-util-3.19.1-4.el7_1.x86_64 numactl-libs-2.0.9-6.el7_2.x86_64 openssl-libs-1.0.2k-19.tl2.1.x86_64 pcre-8.32-15.el7.x86_64 re2-20160401-2.el7.x86_64 sqlite-3.7.17-8.el7.x86_64 xz-libs-5.1.2-12alpha.el7.x86_64 yaml-cpp-0.5.1-2.el7.x86_64 zlib-1.2.7-15.el7.x86_64
(gdb) bt
#0 doNormal16 (mode=CALLBACK_OUTPUT, do_accel=0 '\000', s=1,
end=0x7fa59872864e <Address 0x7fa59872864e out of bounds>, c_inout=, m=0x126eb600)
at /root/hyperscan/src/nfa/mcclellan.c:138
#1 mcclellanExec16_i (mode=CALLBACK_OUTPUT, c_final=0x0, single=0 '\000', ctxt=0x7fa4b03bbbc0,
cb=0x7fa4eae75490 , offAdj=0, len=60,
buf=0x7fa598728612 <Address 0x7fa598728612 out of bounds>, qstate=0x0, state=,
m=0x126eb600) at /root/hyperscan/src/nfa/mcclellan.c:274
#2 nfaExecMcClellan16_Bi (single=0 '\000', context=0x7fa4b03bbbc0,
cb=0x7fa4eae75490 , length=60,
buffer=0x7fa598728612 <Address 0x7fa598728612 out of bounds>, offset=0, n=0x126eb5c0)
at /root/hyperscan/src/nfa/mcclellan.c:763
#3 nfaExecMcClellan16_B (n=0x126eb5c0, offset=0,
buffer=0x7fa598728612 <Address 0x7fa598728612 out of bounds>, length=60,
cb=0x7fa4eae75490 , context=0x7fa4b03bbbc0)
at /root/hyperscan/src/nfa/mcclellan.c:971
#4 0x00007fa4eae625bd in runAnchoredTableBlock (t=, scratch=0x7fa4b03bbbc0,
atable=) at /root/hyperscan/src/rose/block.c:67
#5 roseBlockAnchored (scratch=0x7fa4b03bbbc0, t=0x126d6580) at /root/hyperscan/src/rose/block.c:212
#6 roseBlockExec (t=, scratch=) at /root/hyperscan/src/rose/block.c:395
#7 0x00007fa4ead93f9e in rawBlockExec (scratch=0x7fa4b03bbbc0, rose=0x126d6580)
at /root/hyperscan/src/runtime.c:188
#8 hs_scan (db=, data=, length=2644, flags=,
scratch=0x7fa4b03bbbc0, onEvent=, userCtx=0x7fa4c5e6d290)
at /root/hyperscan/src/runtime.c:419
#9 0x00000000006bed9c in SCHSSearch (mpm_ctx=, mpm_thread_ctx=,
pmq=, buf=, buflen=) at util-mpm-hs.c:938
#10 0x000000000058c04a in StreamMpmFunc (cb_data=, data=,
data_len=) at detect-engine-payload.c:64
#11 0x000000000067af5c in StreamReassembleRawInline (progress_out=0x7fa4b03b8580, cb_data=0x7fa4c5e6db70,
Callback=0x58c010 , p=0x7fa4b032cf60, ssn=)
at stream-tcp-reassemble.c:1487
#12 StreamReassembleRaw (ssn=, p=p@entry=0x7fa4b028be90,
Callback=Callback@entry=0x58c010 , cb_data=cb_data@entry=0x7fa4c5e6db70,
progress_out=progress_out@entry=0x7fa4b03b8580,
respect_inspect_depth=respect_inspect_depth@entry=false) at stream-tcp-reassemble.c:1677
#13 0x000000000058c1e8 in PrefilterPktStream (det_ctx=0x7fa4b03b8530, p=0x7fa4b028be90, pectx=0x4749470)
at detect-engine-payload.c:83
#14 0x000000000058f711 in Prefilter (det_ctx=det_ctx@entry=0x7fa4b03b8530, sgh=0xd415110,
p=p@entry=0x7fa4b028be90, flags=) at detect-engine-prefilter.c:169
#15 0x0000000000557c33 in DetectRunPrefilterPkt (tv=0x9736360, scratch=0x7fa4c5e6dc70, p=0x7fa4b028be90,
det_ctx=0x7fa4b03b8530, de_ctx=0x470a9d0) at detect.c:734
#16 DetectRun (th_v=th_v@entry=0x9736360, de_ctx=, det_ctx=0x7fa4b03b8530,
p=p@entry=0x7fa4b028be90) at detect.c:132
#17 0x0000000000559757 in DetectRun (p=0x7fa4b028be90, det_ctx=, de_ctx=,
th_v=0x9736360) at detect.c:1810
#18 DetectNoFlow (p=, det_ctx=, de_ctx=, tv=)
at detect.c:1810
#19 Detect (tv=tv@entry=0x9736360, p=p@entry=0x7fa4b028be90, data=data@entry=0x7fa4b03b8530,
pq=pq@entry=0x0, postpq=postpq@entry=0x0) at detect.c:1870
#20 0x00000000005eef5b in FlowWorker (tv=0x9736360, p=0x7fa4b028be90, data=0x7fa4b02ab430,
preq=0x5e9bfc0, unused=) at flow-worker.c:346
#21 0x0000000000680e0b in TmThreadsSlotVarRun (tv=tv@entry=0x9736360, p=p@entry=0x7fa4b028be90,
slot=slot@entry=0x5e9d3a0) at tm-threads.c:143
#22 0x0000000000661e2c in TmThreadsSlotProcessPkt (p=0x7fa4b028be90, s=0x5e9d3a0, tv=0x9736360)
at tm-threads.h:147
#23 ReceiveCFWLoop () at source-cfw.c:378
#24 0x0000000000681ee2 in TmThreadsSlotPktAcqLoop (td=0x9736360) at tm-threads.c:346
#25 0x00007fa4e9636e25 in start_thread () from /lib64/libpthread.so.0
---Type to continue, or q to quit---
#26 0x00007fa4e8f4935d in clone () from /lib64/libc.so.6


Files

suricata.yaml (72.7 KB) suricata.yaml eric fool, 07/29/2021 02:28 AM
Actions #1

Updated by eric fool about 2 months ago

Actions #2

Updated by eric fool about 2 months ago

attached file is suricata config. suricata version is 4.1.0. mode is IPS worker. rules is th latest on suricata web site. receiver packets from rte_ring added by myself. Traffics is produced by T-rex, with http connects of 2000 cps and get 66636Bytes per connect.
cpu info:
[root@VM-0-49-centos ~/txfw]# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU: 16
On-line CPU list: 0-15
Thread(s) per core: 1
Core(s) per socket: 8
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz
Stepping: 5
CPU MHz: 2494.134
BogoMIPS: 4988.26
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 4096K
L3 cache: 36608K
NUMA node0 CPU: 0-7
NUMA node1 CPU: 8-15
OS: centos 4.14.105 on VM of KVM

Actions #3

Updated by Victor Julien about 2 months ago

"suricata version is 4.1.0"

This is very old. Did you mean 4.1.10? Regardless the 4.1 branch is EOL. Do you see the same issue with 5.0.7 or 6.0.3?

Actions

Also available in: Atom PDF