Suricata

Hello,

Thanks for the report. This bug should have been fixed in latest git by commit https://github.com/inliniac/suricata/commit/4a1a008009563f12e995eb1f01dd0bdd4f3c62de. I should have put the traceback in the commit message to help you find it.

OK thanks, Eric. I should have tried the latest from git as a matter of course anyway.

I pulled from git this morning. It runs for a while but still segfaults eventually. Output from a couple of different runs below. I'm not a developer at all, so please educate me if I'm not giving you the info you need or missing some other part of the proper protocol to submit bugs.

Cheers,
Michael

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb4842b70 (LWP 4757)]
0x080a8b67 in DetectEngineRunHttpHeaderMpm (det_ctx=0xc458ce8, f=0x7e216588, htp_state=0xbbef0cc0, flags=4 '\004')
    at detect-engine-hhd.c:198
198                 cnt += HttpHeaderPatternSearch(det_ctx,
(gdb) bt
#0  0x080a8b67 in DetectEngineRunHttpHeaderMpm (det_ctx=0xc458ce8, f=0x7e216588, htp_state=0xbbef0cc0, flags=4 '\004')
    at detect-engine-hhd.c:198
#1  0x0807f18f in DetectMpmPrefilter (th_v=0xaf05740, de_ctx=0x9c4cf00, det_ctx=0xc458ce8, p=0x91a2028)
    at detect.c:1282
#2  SigMatchSignatures (th_v=0xaf05740, de_ctx=0x9c4cf00, det_ctx=0xc458ce8, p=0x91a2028) at detect.c:1596
#3  0x08081386 in Detect (tv=0xaf05740, p=0x91a2028, data=0x0, pq=0xb94e538, postpq=0x0) at detect.c:2029
#4  0x08165aaf in TmThreadsSlotVarRun (tv=0xaf05740, p=0x91a2028, slot=0xb94e518) at tm-threads.c:524
#5  0x08165de2 in TmThreadsSlotVar (td=0xaf05740) at tm-threads.c:749
#6  0x001ac96e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#7  0x002dca4e in clone () from /lib/tls/i686/cmov/libc.so.6

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb4842b70 (LWP 13314)]
SCProfilingAddPacket (p=0x55e539f8) at util-profiling.c:680
680             if (PKT_IS_IPV4(p)) {
(gdb) bt
#0  SCProfilingAddPacket (p=0x55e539f8) at util-profiling.c:680
#1  0x08168b84 in TmqhOutputPacketpool (t=0xaf05740, p=0x55e539f8) at tmqh-packetpool.c:204
#2  0x08165df5 in TmThreadsSlotVar (td=0xaf05740) at tm-threads.c:757
#3  0x001ac96e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#4  0x002dca4e in clone () from /lib/tls/i686/cmov/libc.so.6

Hi Michael,

Can you paste the output of

print det_ctx->hhd_buffers_list_len

and for every i from 0 upto (det_ctx->hhd_buffers_list_len - 1)

print det_ctx->hhd_buffers[i]
print det_ctx->hhd_buffers_len[i]

See below for output. I hope I did it correctly; I only have a vague idea of what I'm looking at.

To answer some of the questions in the email from Peter, this is a busy sensor. The NIC sees ~400-500mbps, but I'm really only trying to inspect ~200mbps of gre-encapsulated proxy traffic. The hardware is a bit overloaded, but it's what I have to test with at the moment. I bumped up all the memcap lines in the yaml, but I'm not running out of physical RAM. Everything else is default except for ip vars and disabling some rules. It will run for a varying amount of time, maybe 10-40 minutes.

Hope that helps.

Regards,
Michael

[6760] 25/9/2012 -- 16:45:59 - (tm-threads.c:2054) <Info> (TmThreadWaitOnThreadInit) -- all 7 packet processing threads, 3 management threads initialized, engine started.
[7653] 25/9/2012 -- 16:45:59 - (source-af-packet.c:1322) <Info> (AFPCreateSocket) -- Using interface 'eth1' via socket 9
[7653] 25/9/2012 -- 16:45:59 - (source-af-packet.c:412) <Info> (AFPPeersListReachedInc) -- All AFP capture threads are running.
[7653] 25/9/2012 -- 16:45:59 - (source-af-packet.c:938) <Info> (ReceiveAFPLoop) -- Thread RxAFP1 using socket 9

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb09b8b70 (LWP 7658)]
0x080a8b67 in DetectEngineRunHttpHeaderMpm (det_ctx=0xd417338, f=0x19f2e240, htp_state=0x43c15010, flags=6 '\006')
    at detect-engine-hhd.c:198
198                 cnt += HttpHeaderPatternSearch(det_ctx,
(gdb) bt
#0  0x080a8b67 in DetectEngineRunHttpHeaderMpm (det_ctx=0xd417338, f=0x19f2e240, htp_state=0x43c15010, flags=6 '\006')
    at detect-engine-hhd.c:198
#1  0x0807f18f in DetectMpmPrefilter (th_v=0xb8dcec8, de_ctx=0x9528348, det_ctx=0xd417338, p=0x900fd90)
    at detect.c:1282
#2  SigMatchSignatures (th_v=0xb8dcec8, de_ctx=0x9528348, det_ctx=0xd417338, p=0x900fd90) at detect.c:1596
#3  0x08081386 in Detect (tv=0xb8dcec8, p=0x900fd90, data=0x0, pq=0xcc03328, postpq=0x0) at detect.c:2029
#4  0x08165aaf in TmThreadsSlotVarRun (tv=0xb8dcec8, p=0x900fd90, slot=0xcc03308) at tm-threads.c:524
#5  0x08165de2 in TmThreadsSlotVar (td=0xb8dcec8) at tm-threads.c:749
#6  0x001ac96e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#7  0x002dca4e in clone () from /lib/tls/i686/cmov/libc.so.6

(gdb) print det_ctx->hhd_buffers_list_len
$1 = 1
(gdb) print det_ctx->hhd_buffers[0]
$2 = (uint8_t *) 0x0
(gdb) print det_ctx->hhd_buffers_len[0]
Cannot access memory at address 0x0

Thanks. Can you reproduce this frequently?

Every run so far (5 or 6). I'll do some more tests this afternoon if possible and also look for a different location to test with a different mix of traffic and hardware.

I have my fix in this branch - https://github.com/poona/suricata/tree/task_29_bug_557

Can you pull this branch and test it?

OR

you can download the patch and apply it directly on your branch

Patch: https://github.com/poona/suricata/commit/f983fefdf5385bdf5dc281acb3bfb00cc27cc000.patch

With the new branch.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb5ab1b70 (LWP 14986)]
0x0812056c in SCACSearch (mpm_ctx=0xbb521d0, mpm_thread_ctx=0xd7e6e04, pmq=0xd7e6e1c, buf=0x0, buflen=180)
    at util-mpm-ac.c:1232
1232                state = state_table_u16[state & 0x7FFF][u8_tolower(buf[i])];
(gdb) bt
#0  0x0812056c in SCACSearch (mpm_ctx=0xbb521d0, mpm_thread_ctx=0xd7e6e04, pmq=0xd7e6e1c, buf=0x0, buflen=180)
    at util-mpm-ac.c:1232
#1  0x0809b060 in HttpHeaderPatternSearch (det_ctx=0xd7e6db0, headers=0x0, headers_len=180, flags=8 '\b')
    at detect-engine-mpm.c:393
#2  0x080a8e17 in DetectEngineRunHttpHeaderMpm (det_ctx=0xd7e6db0, f=0x167d3888, htp_state=0x5888fb18, flags=8 '\b')
    at detect-engine-hhd.c:201
#3  0x0807f3ff in DetectMpmPrefilter (th_v=0xb8cc370, de_ctx=0x9529348, det_ctx=0xd7e6db0, p=0x904a670)
    at detect.c:1286
#4  SigMatchSignatures (th_v=0xb8cc370, de_ctx=0x9529348, det_ctx=0xd7e6db0, p=0x904a670) at detect.c:1600
#5  0x080815f6 in Detect (tv=0xb8cc370, p=0x904a670, data=0x364ff4, pq=0xd7328f8, postpq=0x0) at detect.c:2033
#6  0x08165c3f in TmThreadsSlotVarRun (tv=0xb8cc370, p=0x904a670, slot=0xd7328d8) at tm-threads.c:524
#7  0x08165f72 in TmThreadsSlotVar (td=0xb8cc370) at tm-threads.c:749
#8  0x001ac96e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#9  0x002dca4e in clone () from /lib/tls/i686/cmov/libc.so.6

Should fix it.

https://github.com/poona/suricata/commit/a9946ea7c3bc5733ba828b5eae164e61fecebcda.patch

The patch supplied so far should fix the segv seen so far, but there are more issues <- all of these under low memory conditions. Fixing them

pull request here - https://github.com/inliniac/suricata/pull/103

The memory alloc handling in libhtp needs to be fixed though. You would still see segvs because of this.

Thanks, Anoop (and all for working on a great project!).

I'll make sure I give myself plenty of memory headroom so that I can continue to test the 1.4dev.

Please note that on 32bit you'll be limited to ~3GB per process no matter what you do. So suricata will not be able to use more. I recommend a 64 bit setup for your sensor.