Project

General

Profile

Actions

Bug #2888

closed

4.1.3 core in HCBDCreateSpace

Added by Andy Wick almost 4 years ago. Updated almost 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

Getting many cores

#0  0x00007fe6de5a5207 in raise () from /lib64/libc.so.6
#1  0x00007fe6de5a68f8 in abort () from /lib64/libc.so.6
#2  0x00007fe6de5e7d27 in __libc_message () from /lib64/libc.so.6
#3  0x00007fe6de5ee5d4 in malloc_printerr () from /lib64/libc.so.6
#4  0x00007fe6de5f49e9 in realloc () from /lib64/libc.so.6
#5  0x00000000004b3b80 in HCBDCreateSpace (det_ctx=0x7fe695be5a50, size=<optimized out>) at detect-engine-hcbd.c:80
#6  0x00000000004b3ea7 in DetectEngineHCBDGetBufferForTX (tx=0x7fe696d6d370, tx_id=184, det_ctx=det_ctx@entry=0x7fe695be5a50, flags=flags@entry=132 '\204', buffer_len=buffer_len@entry=0x7fe69fffd738,
stream_start_offset=stream_start_offset@entry=0x7fe69fffd73c, htp_state=<optimized out>, f=<optimized out>, de_ctx=0x0) at detect-engine-hcbd.c:140
#7  0x00000000004b40a2 in PrefilterTxHttpRequestBody (det_ctx=0x7fe695be5a50, pectx=0x7fe6c320d300, p=<optimized out>, f=<optimized out>, txv=<optimized out>, idx=<optimized out>, flags=132 '\204')
at detect-engine-hcbd.c:241
#8  0x00000000004c32f1 in DetectRunPrefilterTx (det_ctx=det_ctx@entry=0x7fe695be5a50, sgh=sgh@entry=0x7fe6c301d640, p=p@entry=0x7fe695afed50, ipproto=ipproto@entry=6 '\006', flow_flags=flow_flags@entry=132 '\204',
alproto=alproto@entry=1, alstate=alstate@entry=0x7fe6969a1830, tx=tx@entry=0x7fe69fffd960) at detect-engine-prefilter.c:117
#9  0x000000000048a67a in DetectRunTx (scratch=0x7fe69fffd920, f=0x7fe60f0aebc0, p=0x8, det_ctx=0x7fe695be5a50, de_ctx=0x2c80680, tv=0x7fe6cf339b10) at detect.c:1398
#10 DetectRun (th_v=th_v@entry=0x7fe6cf339b10, de_ctx=0x2c80680, det_ctx=0x7fe695be5a50, p=p@entry=0x7fe695afed50) at detect.c:141
#11 0x000000000048b843 in DetectRun (p=0x7fe695afed50, det_ctx=<optimized out>, de_ctx=<optimized out>, th_v=0x7fe6cf339b10) at detect.c:1641
#12 DetectNoFlow (p=<optimized out>, det_ctx=<optimized out>, de_ctx=<optimized out>, tv=<optimized out>) at detect.c:1679
#13 Detect (tv=tv@entry=0x7fe6cf339b10, p=p@entry=0x7fe695afed50, data=data@entry=0x7fe695be5a50, pq=pq@entry=0x0, postpq=postpq@entry=0x0) at detect.c:1739
#14 0x000000000051cccb in FlowWorker (tv=0x7fe6cf339b10, p=0x7fe695afed50, data=0x7fe695b13be0, preq=0x7fe6cf36bc90, unused=<optimized out>) at flow-worker.c:260
#15 0x000000000059accd in TmThreadsSlotVarRun (tv=tv@entry=0x7fe6cf339b10, p=p@entry=0x7fe695afed50, slot=slot@entry=0x3a31f930) at tm-threads.c:145
#16 0x0000000000575c3e in TmThreadsSlotProcessPkt (p=0x7fe695afed50, s=0x3a31f930, tv=0x7fe6cf339b10) at tm-threads.h:147
#17 AFPReadFromRing (ptv=0x7fe695aff700) at source-af-packet.c:1016
#18 0x0000000000578fbe in ReceiveAFPLoop (tv=0x7fe6cf339b10, data=0x7fe695aff700, slot=<optimized out>) at source-af-packet.c:1579
#19 0x000000000059d432 in TmThreadsSlotPktAcqLoop (td=0x7fe6cf339b10) at tm-threads.c:348
#20 0x00007fe6df1f1dd5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007fe6de66cead in clone () from /lib64/libc.so.6

This valgrind maybe is related

==21521== Thread 12 W#11-ens5f1:
==21521== Invalid write of size 8
==21521==    at 0x5FF596: StreamingBufferGetDataAtOffset (util-streaming-buffer.c:875)
==21521==    by 0x4B3FCD: DetectEngineHCBDGetBufferForTX.isra.1 (detect-engine-hcbd.c:203)
==21521==    by 0x4B40A1: PrefilterTxHttpRequestBody (detect-engine-hcbd.c:241)
==21521==    by 0x4C32F0: DetectRunPrefilterTx (detect-engine-prefilter.c:117)
==21521==    by 0x48A679: DetectRunTx (detect.c:1398)
==21521==    by 0x48A679: DetectRun.part.19 (detect.c:141)
==21521==    by 0x48B842: DetectRun (detect.c:1641)
==21521==    by 0x48B842: DetectNoFlow (detect.c:1679)
==21521==    by 0x48B842: Detect (detect.c:1739)
==21521==    by 0x51CCCA: FlowWorker (flow-worker.c:260)
==21521==    by 0x59ACCC: TmThreadsSlotVarRun (tm-threads.c:145)
==21521==    by 0x575C8E: TmThreadsSlotProcessPkt (tm-threads.h:176)
==21521==    by 0x575C8E: AFPReadFromRing (source-af-packet.c:1016)
==21521==    by 0x578FBD: ReceiveAFPLoop (source-af-packet.c:1579)
==21521==    by 0x59D431: TmThreadsSlotPktAcqLoop (tm-threads.c:348)
==21521==    by 0x6044DD4: start_thread (in /usr/lib64/libpthread-2.17.so)
==21521==  Address 0xa047b0f8 is 24 bytes after a block of size 2,464 in arena "client" 
==21521==
==21521== Invalid write of size 4
==21521==    at 0x5FF599: StreamingBufferGetDataAtOffset (util-streaming-buffer.c:876)
==21521==    by 0x4B3FCD: DetectEngineHCBDGetBufferForTX.isra.1 (detect-engine-hcbd.c:203)
==21521==    by 0x4B40A1: PrefilterTxHttpRequestBody (detect-engine-hcbd.c:241)
==21521==    by 0x4C32F0: DetectRunPrefilterTx (detect-engine-prefilter.c:117)
==21521==    by 0x48A679: DetectRunTx (detect.c:1398)
==21521==    by 0x48A679: DetectRun.part.19 (detect.c:141)
==21521==    by 0x48B842: DetectRun (detect.c:1641)
==21521==    by 0x48B842: DetectNoFlow (detect.c:1679)
==21521==    by 0x48B842: Detect (detect.c:1739)
==21521==    by 0x51CCCA: FlowWorker (flow-worker.c:260)
==21521==    by 0x59ACCC: TmThreadsSlotVarRun (tm-threads.c:145)
==21521==    by 0x575C8E: TmThreadsSlotProcessPkt (tm-threads.h:176)
==21521==    by 0x575C8E: AFPReadFromRing (source-af-packet.c:1016)
==21521==    by 0x578FBD: ReceiveAFPLoop (source-af-packet.c:1579)
==21521==    by 0x59D431: TmThreadsSlotPktAcqLoop (tm-threads.c:348)
==21521==    by 0x6044DD4: start_thread (in /usr/lib64/libpthread-2.17.so)
==21521==  Address 0xa047b10c is 20 bytes before a block of size 2,000 alloc'd
==21521==    at 0x4C29B0D: malloc (vg_replace_malloc.c:298)
==21521==    by 0x4C2BAD9: realloc (vg_replace_malloc.c:785)
==21521==    by 0x4B3B7F: HCBDCreateSpace (detect-engine-hcbd.c:80)
==21521==    by 0x4B3EF3: DetectEngineHCBDGetBufferForTX.isra.1 (detect-engine-hcbd.c:125)
==21521==    by 0x4B40A1: PrefilterTxHttpRequestBody (detect-engine-hcbd.c:241)
==21521==    by 0x4C32F0: DetectRunPrefilterTx (detect-engine-prefilter.c:117)
==21521==    by 0x48A679: DetectRunTx (detect.c:1398)
==21521==    by 0x48A679: DetectRun.part.19 (detect.c:141)
==21521==    by 0x48B842: DetectRun (detect.c:1641)
==21521==    by 0x48B842: DetectNoFlow (detect.c:1679)
==21521==    by 0x48B842: Detect (detect.c:1739)
==21521==    by 0x51CCCA: FlowWorker (flow-worker.c:260)
==21521==    by 0x59ACCC: TmThreadsSlotVarRun (tm-threads.c:145)
==21521==    by 0x575C3D: TmThreadsSlotProcessPkt (tm-threads.h:147)
==21521==    by 0x575C3D: AFPReadFromRing (source-af-packet.c:1016)
==21521==    by 0x578FBD: ReceiveAFPLoop (source-af-packet.c:1579)
==21521==
==21521== Invalid write of size 8
==21521==    at 0x4B3FDE: DetectEngineHCBDGetBufferForTX.isra.1 (detect-engine-hcbd.c:206)
==21521==    by 0x4B40A1: PrefilterTxHttpRequestBody (detect-engine-hcbd.c:241)
==21521==    by 0x4C32F0: DetectRunPrefilterTx (detect-engine-prefilter.c:117)
==21521==    by 0x48A679: DetectRunTx (detect.c:1398)
==21521==    by 0x48A679: DetectRun.part.19 (detect.c:141)
==21521==    by 0x48B842: DetectRun (detect.c:1641)
==21521==    by 0x48B842: DetectNoFlow (detect.c:1679)
==21521==    by 0x48B842: Detect (detect.c:1739)
==21521==    by 0x51CCCA: FlowWorker (flow-worker.c:260)
==21521==    by 0x59ACCC: TmThreadsSlotVarRun (tm-threads.c:145)
==21521==    by 0x575C8E: TmThreadsSlotProcessPkt (tm-threads.h:176)
==21521==    by 0x575C8E: AFPReadFromRing (source-af-packet.c:1016)
==21521==    by 0x578FBD: ReceiveAFPLoop (source-af-packet.c:1579)
==21521==    by 0x59D431: TmThreadsSlotPktAcqLoop (tm-threads.c:348)
==21521==    by 0x6044DD4: start_thread (in /usr/lib64/libpthread-2.17.so)
==21521==    by 0x6C04EAC: clone (in /usr/lib64/libc-2.17.so)
==21521==  Address 0xa047b118 is 8 bytes before a block of size 2,000 alloc'd
==21521==    at 0x4C29B0D: malloc (vg_replace_malloc.c:298)
==21521==    by 0x4C2BAD9: realloc (vg_replace_malloc.c:785)
==21521==    by 0x4B3B7F: HCBDCreateSpace (detect-engine-hcbd.c:80)
==21521==    by 0x4B3EF3: DetectEngineHCBDGetBufferForTX.isra.1 (detect-engine-hcbd.c:125)
==21521==    by 0x4B40A1: PrefilterTxHttpRequestBody (detect-engine-hcbd.c:241)
==21521==    by 0x4C32F0: DetectRunPrefilterTx (detect-engine-prefilter.c:117)
==21521==    by 0x48A679: DetectRunTx (detect.c:1398)
==21521==    by 0x48A679: DetectRun.part.19 (detect.c:141)
==21521==    by 0x48B842: DetectRun (detect.c:1641)
==21521==    by 0x48B842: DetectNoFlow (detect.c:1679)
==21521==    by 0x48B842: Detect (detect.c:1739)
==21521==    by 0x51CCCA: FlowWorker (flow-worker.c:260)
==21521==    by 0x59ACCC: TmThreadsSlotVarRun (tm-threads.c:145)
==21521==    by 0x575C3D: TmThreadsSlotProcessPkt (tm-threads.h:147)
==21521==    by 0x575C3D: AFPReadFromRing (source-af-packet.c:1016)
==21521==    by 0x578FBD: ReceiveAFPLoop (source-af-packet.c:1579)
==21521==


Related issues 1 (0 open1 closed)

Related to Bug #2936: Several crashes past week Suricata 4.1.3 , last : double free or corruptionClosedActions
Actions #1

Updated by Peter Manev almost 4 years ago

Thank you Andy for the bug report!
Could you please share some details how to reproduce this with a pcap it is possible ?

Actions #2

Updated by Andy Wick almost 4 years ago

Unfortunately running against live traffic so not sure if I'll be able to find a pcap or not. I'll see what I can find.

Actions #3

Updated by Andy Wick almost 4 years ago

Not sure if this helps but

index is -1 for detect-engine-hcbd.c:203

    StreamingBufferGetDataAtOffset(htud->request_body.sb,
            &det_ctx->hcbd[index].buffer, &det_ctx->hcbd[index].buffer_len,
            offset);

here is other gdb stuff that might help

(gdb) p index
$12 = -1
(gdb) p *det_ctx
$13 = {tenant_id = 0, ticker = 1139548, tv = 0xb9f09a0, non_pf_id_array = 0x621a8630, non_pf_id_cnt = 3, mt_det_ctxs_cnt = 0, mt_det_ctxs = 0x0, mt_det_ctxs_hash = 0x0, tenant_array = 0x0,
  tenant_array_size = 0, TenantGetId = 0x0, raw_stream_progress = 17889, buffer_offset = 0, pcre_match_start_offset = 0, filestore_cnt = 0, hcbd = 0x9de88f90, hcbd_start_tx_id = 6,
  hcbd_buffers_size = 50, hcbd_buffers_list_len = 0, counter_alerts = 172, inspect_list = 0, inspect = {buffers = 0xeb7fb4c0, buffers_size = 67, to_clear_idx = 1,
    to_clear_queue = 0x94cf29b0}, multi_inspect = {buffers = 0xa0175970, buffers_size = 67, to_clear_idx = 0, to_clear_queue = 0x9647c8c0}, discontinue_matching = 1, flags = 0,
  tx_id_set = 0, tx_id = 0, p = 0x0, so_far_used_by_detect_sc_atomic__ = 1, inspection_recursion_counter = 1, match_array = 0xeb6cbed0, match_array_len = 31062, match_array_cnt = 8,
  tx_candidates = 0xeb7089c0, tx_candidates_size = 31062, non_pf_store_ptr = 0xa46b8990, non_pf_store_cnt = 5, mtc = {ctx = 0xb968de0, memory_cnt = 1, memory_size = 16}, mtcu = {
    ctx = 0xbc9f6e0, memory_cnt = 1, memory_size = 16}, mtcs = {ctx = 0x8a30dd0, memory_cnt = 1, memory_size = 16}, pmq = {rule_id_array = 0x14eabc170, rule_id_array_cnt = 2,
    rule_id_array_size = 3590}, spm_thread_ctx = 0x62111b20, io_ctx = {sig_match_array = 0xeb6caf70 "", sig_match_size = 3860}, bj_values = 0x621a86c0, replist = 0x0, varlist = 0x0,
  filestore = {{file_id = 0, tx_id = 0} <repeats 15 times>}, de_ctx = 0xb35c3f0, keyword_ctxs_array = 0x0, keyword_ctxs_size = 0, global_keyword_ctxs_size = 3,
  global_keyword_ctxs_array = 0x5770b2e0, base64_decoded = 0x0, base64_decoded_len = 0, base64_decoded_len_max = 0, decoder_events = 0x0, events = 0}
(gdb) p *htud
$14 = {detect_flags_ts = 0, detect_flags_tc = 0, request_body_init = 1 '\001', response_body_init = 1 '\001', request_has_trailers = 0 '\000', response_has_trailers = 0 '\000', logged = 0,
  request_body = {first = 0x14eadd110, last = 0x14eadd110, sb = 0x5b142d40, content_len_so_far = 2659, body_parsed = 0, body_inspected = 0}, response_body = {first = 0x1bde7d70,
    last = 0x1bde7d70, sb = 0x1bde7d00, content_len_so_far = 146, body_parsed = 146, body_inspected = 0}, request_uri_normalized = 0x9919aac0,
  request_headers_raw = 0x442162e0 "REDACTED"...,
  response_headers_raw = 0xb24e000 "Content-Type: application/json\r\nCache-Control: private, no-cache, no-store, must-revalidate, max-age=0\r\nP3P: policyref=\"REDACTED", CP=\"CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi"..., request_headers_raw_len = 1168, response_headers_raw_len = 401, decoder_events = 0x0, boundary = 0x0, boundary_len = 0 '\000',
  tsflags = 0 '\000', tcflags = 0 '\000', request_body_type = 2 '\002', de_state = 0x0}

Actions #4

Updated by Victor Julien almost 4 years ago

Thanks Andy. The gdb output in note number #3, is that from the same crash as the bt above? If not, could you produce both outputs for a single crash?

Actions #5

Updated by Andy Wick almost 4 years ago

I'm getting lots of different crashes. #3 was from valgrind break on error of the valgrind stuff in #1. I worry that memory is already corrupt for the bt of #1. (Otherwise why would a realloc fail?) Anyway here is another crash with some vars printed:

(gdb) bt
#0  0x00007f3ba486b207 in raise () from /lib64/libc.so.6
#1  0x00007f3ba486c8f8 in abort () from /lib64/libc.so.6
#2  0x00007f3ba48add27 in __libc_message () from /lib64/libc.so.6
#3  0x00007f3ba48b45d4 in malloc_printerr () from /lib64/libc.so.6
#4  0x00007f3ba48ba9e9 in realloc () from /lib64/libc.so.6
#5  0x00000000004b3b80 in HCBDCreateSpace (det_ctx=0x7f3b69be8500, size=<optimized out>) at detect-engine-hcbd.c:80
#6  0x00000000004b3ea7 in DetectEngineHCBDGetBufferForTX (tx=0x7f3a88f26940, tx_id=158, det_ctx=det_ctx@entry=0x7f3b69be8500, flags=flags@entry=132 '\204',
    buffer_len=buffer_len@entry=0x7f3b7282c738, stream_start_offset=stream_start_offset@entry=0x7f3b7282c73c, htp_state=<optimized out>, f=<optimized out>, de_ctx=0x0)
    at detect-engine-hcbd.c:140
#7  0x00000000004b40a2 in PrefilterTxHttpRequestBody (det_ctx=0x7f3b69be8500, pectx=0x7f3b88c4c0e0, p=<optimized out>, f=<optimized out>, txv=<optimized out>, idx=<optimized out>,
    flags=132 '\204') at detect-engine-hcbd.c:241
#8  0x00000000004c32f1 in DetectRunPrefilterTx (det_ctx=det_ctx@entry=0x7f3b69be8500, sgh=sgh@entry=0x7f3b88c4f530, p=p@entry=0x7f3b69afed50, ipproto=ipproto@entry=6 '\006',
    flow_flags=flow_flags@entry=132 '\204', alproto=alproto@entry=1, alstate=alstate@entry=0x7f3a88f6c1b0, tx=tx@entry=0x7f3b7282c960) at detect-engine-prefilter.c:117
#9  0x000000000048a67a in DetectRunTx (scratch=0x7f3b7282c920, f=0x7f3aa1aa42b0, p=0x8, det_ctx=0x7f3b69be8500, de_ctx=0x33cf680, tv=0x7f3b94f1d870) at detect.c:1398
#10 DetectRun (th_v=th_v@entry=0x7f3b94f1d870, de_ctx=0x33cf680, det_ctx=0x7f3b69be8500, p=p@entry=0x7f3b69afed50) at detect.c:141
#11 0x000000000048b843 in DetectRun (p=0x7f3b69afed50, det_ctx=<optimized out>, de_ctx=<optimized out>, th_v=0x7f3b94f1d870) at detect.c:1641
#12 DetectNoFlow (p=<optimized out>, det_ctx=<optimized out>, de_ctx=<optimized out>, tv=<optimized out>) at detect.c:1679
#13 Detect (tv=tv@entry=0x7f3b94f1d870, p=p@entry=0x7f3b69afed50, data=data@entry=0x7f3b69be8500, pq=pq@entry=0x0, postpq=postpq@entry=0x0) at detect.c:1739
#14 0x000000000051cccb in FlowWorker (tv=0x7f3b94f1d870, p=0x7f3b69afed50, data=0x7f3b69b165d0, preq=0x7f3b94f53680, unused=<optimized out>) at flow-worker.c:260
#15 0x000000000059accd in TmThreadsSlotVarRun (tv=tv@entry=0x7f3b94f1d870, p=p@entry=0x7f3b69afed50, slot=slot@entry=0x1afd8d80) at tm-threads.c:145
#16 0x0000000000575c3e in TmThreadsSlotProcessPkt (p=0x7f3b69afed50, s=0x1afd8d80, tv=0x7f3b94f1d870) at tm-threads.h:147
#17 AFPReadFromRing (ptv=0x7f3b69aff700) at source-af-packet.c:1016
#18 0x0000000000578fbe in ReceiveAFPLoop (tv=0x7f3b94f1d870, data=0x7f3b69aff700, slot=<optimized out>) at source-af-packet.c:1579
#19 0x000000000059d432 in TmThreadsSlotPktAcqLoop (td=0x7f3b94f1d870) at tm-threads.c:348
#20 0x00007f3ba54b7dd5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007f3ba4932ead in clone () from /lib64/libc.so.6
(gdb) p *det_ctx
$2 = {tenant_id = 0, ticker = 831308, tv = 0x7f3b94f1d870, non_pf_id_array = 0x7f3b69be8a40, non_pf_id_cnt = 3, mt_det_ctxs_cnt = 0, mt_det_ctxs = 0x0, mt_det_ctxs_hash = 0x0,
  tenant_array = 0x0, tenant_array_size = 0, TenantGetId = 0x0, raw_stream_progress = 68649, buffer_offset = 5, pcre_match_start_offset = 580, filestore_cnt = 0,
  hcbd = 0x7f3b6a7e6bd0, hcbd_start_tx_id = 6, hcbd_buffers_size = 100, hcbd_buffers_list_len = 1, counter_alerts = 172, inspect_list = 0, inspect = {buffers = 0x7f3b69d18f70,
    buffers_size = 67, to_clear_idx = 1, to_clear_queue = 0x7f3b69d19e20}, multi_inspect = {buffers = 0x7f3b69d19f40, buffers_size = 67, to_clear_idx = 0,
    to_clear_queue = 0x7f3b69d1a380}, discontinue_matching = 1, flags = 0, tx_id_set = 0, tx_id = 0, p = 0x0, so_far_used_by_detect_sc_atomic__ = 1, inspection_recursion_counter = 2,
  match_array = 0x7f3b69be99b0, match_array_len = 31062, match_array_cnt = 8, tx_candidates = 0x7f3b69c26470, tx_candidates_size = 31062, non_pf_store_ptr = 0x7f3b8797b1d0,
  non_pf_store_cnt = 5, mtc = {ctx = 0x7f3b69be87b0, memory_cnt = 1, memory_size = 16}, mtcu = {ctx = 0x7f3b69be87f0, memory_cnt = 1, memory_size = 16}, mtcs = {ctx = 0x7f3b69be87d0,
    memory_cnt = 1, memory_size = 16}, pmq = {rule_id_array = 0x7f3b6abf0ea0, rule_id_array_cnt = 2, rule_id_array_size = 3590}, spm_thread_ctx = 0x7f3b69be8a20, io_ctx = {
    sig_match_array = 0x7f3b69be8a90 "", sig_match_size = 3860}, bj_values = 0x7f3b69d18f40, replist = 0x0, varlist = 0x0, filestore = {{file_id = 0, tx_id = 0} <repeats 15 times>},
  de_ctx = 0x33cf680, keyword_ctxs_array = 0x0, keyword_ctxs_size = 0, global_keyword_ctxs_size = 3, global_keyword_ctxs_array = 0x7f3b69d1a4a0, base64_decoded = 0x0,
  base64_decoded_len = 0, base64_decoded_len_max = 0, decoder_events = 0x0, events = 0}
(gdb) p grow_by
$3 = 53
(gdb) p size
$4 = <optimized out>
(gdb) up
#6  0x00000000004b3ea7 in DetectEngineHCBDGetBufferForTX (tx=0x7f3a88f26940, tx_id=158, det_ctx=det_ctx@entry=0x7f3b69be8500, flags=flags@entry=132 '\204',
    buffer_len=buffer_len@entry=0x7f3b7282c738, stream_start_offset=stream_start_offset@entry=0x7f3b7282c73c, htp_state=<optimized out>, f=<optimized out>, de_ctx=0x0)
    at detect-engine-hcbd.c:140
140    in detect-engine-hcbd.c
(gdb) p txs
$5 = 153

and here is an example where I think its showing memory is already corrupt

(gdb) bt
#0  0x00007f3a1b98028a in _int_free () from /lib64/libc.so.6
#1  0x0000000000590839 in ReassembleFree (ptr=<optimized out>, size=2048) at stream-tcp-reassemble.c:237
#2  0x00000000005fe97f in StreamingBufferClear (sb=sb@entry=0x7f39e30f10f8) at util-streaming-buffer.c:139
#3  0x0000000000584f00 in StreamTcpStreamCleanup (stream=stream@entry=0x7f39e30f10c0) at stream-tcp.c:207
#4  0x0000000000584f2a in StreamTcpSessionCleanup (ssn=ssn@entry=0x7f39e30f1030) at stream-tcp.c:223
#5  0x0000000000584f8e in StreamTcpSessionClear (ssnptr=0x7f39e30f1030) at stream-tcp.c:256
#6  0x000000000051687e in FlowClearMemory (f=f@entry=0x7f394b6c0180, proto_map=<optimized out>) at flow.c:965
#7  0x000000000051882e in FlowRecycler (th_v=0x7f3a0b4b8ac0, thread_data=0x7f393c0008c0) at flow-manager.c:923
#8  0x000000000059b9b7 in TmThreadsManagement (td=0x7f3a0b4b8ac0) at tm-threads.c:719
#9  0x00007f3a1c581dd5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f3a1b9fcead in clone () from /lib64/libc.so.6

Does suricata make it easy to compile with asan, maybe that would help?

Actions #6

Updated by Victor Julien almost 4 years ago

Did you run 4.1.2 before? Does that show the issue as well?

I agree this looks like mem corruption.

Actions #7

Updated by Andy Wick almost 4 years ago

We were running 4.0.1 so made a big jump. Should we try 4.1.2?

Actions #8

Updated by Victor Julien almost 4 years ago

Are you able to compile Suricata 4.1.3 with ASAN enabled? Then it should blow up at the first error, not at some later time.

Actions #9

Updated by Andy Wick almost 4 years ago

Sure, thats what I was asking before, is it easy to do? Pointers how?

Actions #10

Updated by Victor Julien almost 4 years ago

If you compiler is recent enough, you should just be able to add '-fsanitize=address -fno-omit-frame-pointer' to your CFLAGS.

Actions #11

Updated by Andy Wick almost 4 years ago

4.1.2 has been up over 45 minutes with no cores, compared to a few minutes that 4.1.3 would last before coring. I'll look at getting 4.1.3 recompiled with ASAN

Actions #12

Updated by Victor Julien almost 4 years ago

Hi Andy, have you seen no crashes with 4.1.2 still?

Actions #13

Updated by Andy Wick almost 4 years ago

Yes, still no cores on 4.1.2. I still owe your 4.1.3 compiled with ASAN

Actions #14

Updated by Andy Wick almost 4 years ago

=================================================================
==31185==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62d0078003d8 at pc 0x0000008eb257 bp 0x7f478992c300 sp 0x7f478992c2f0
WRITE of size 8 at 0x62d0078003d8 thread T1 (W#01-ens5f1)
    #0 0x8eb256 in StreamingBufferGetDataAtOffset /root/suricata-4.1.3/src/util-streaming-buffer.c:875
    #1 0x5cff41 in DetectEngineHCBDGetBufferForTX /root/suricata-4.1.3/src/detect-engine-hcbd.c:203
    #2 0x5d0564 in PrefilterTxHttpRequestBody /root/suricata-4.1.3/src/detect-engine-hcbd.c:241
    #3 0x5f4f9d in DetectRunPrefilterTx /root/suricata-4.1.3/src/detect-engine-prefilter.c:117
    #4 0x567599 in DetectRunTx /root/suricata-4.1.3/src/detect.c:1398
    #5 0x567599 in DetectRun /root/suricata-4.1.3/src/detect.c:141
    #6 0x569fda in DetectNoFlow /root/suricata-4.1.3/src/detect.c:1679
    #7 0x569fda in Detect /root/suricata-4.1.3/src/detect.c:1739
    #8 0x6c39b4 in FlowWorker /root/suricata-4.1.3/src/flow-worker.c:260
    #9 0x801f1e in TmThreadsSlotVarRun /root/suricata-4.1.3/src/tm-threads.c:145
    #10 0x78ed09 in TmThreadsSlotProcessPkt /root/suricata-4.1.3/src/tm-threads.h:147
    #11 0x78ed09 in AFPReadFromRing /root/suricata-4.1.3/src/source-af-packet.c:1016
    #12 0x7982db in ReceiveAFPLoop /root/suricata-4.1.3/src/source-af-packet.c:1579
    #13 0x804b93 in TmThreadsSlotPktAcqLoop /root/suricata-4.1.3/src/tm-threads.c:348
    #14 0x7f47988dddd4 in start_thread (/lib64/libpthread.so.0+0x7dd4)
    #15 0x7f4797d58eac in __clone (/lib64/libc.so.6+0xfdeac)

0x62d0078003d8 is located 40 bytes to the left of 34000-byte region [0x62d007800400,0x62d0078088d0)
allocated by thread T1 (W#01-ens5f1) here:
    #0 0x7f4799dd8c90 in realloc (/lib64/libasan.so.4+0xdec90)
    #1 0x5cf572 in HCBDCreateSpace /root/suricata-4.1.3/src/detect-engine-hcbd.c:80
    #2 0x5d0119 in DetectEngineHCBDGetBufferForTX /root/suricata-4.1.3/src/detect-engine-hcbd.c:125
    #3 0x5d0564 in PrefilterTxHttpRequestBody /root/suricata-4.1.3/src/detect-engine-hcbd.c:241
    #4 0x5f4f9d in DetectRunPrefilterTx /root/suricata-4.1.3/src/detect-engine-prefilter.c:117
    #5 0x567599 in DetectRunTx /root/suricata-4.1.3/src/detect.c:1398
    #6 0x567599 in DetectRun /root/suricata-4.1.3/src/detect.c:141
    #7 0x569fda in DetectNoFlow /root/suricata-4.1.3/src/detect.c:1679
    #8 0x569fda in Detect /root/suricata-4.1.3/src/detect.c:1739
    #9 0x6c39b4 in FlowWorker /root/suricata-4.1.3/src/flow-worker.c:260
    #10 0x801f1e in TmThreadsSlotVarRun /root/suricata-4.1.3/src/tm-threads.c:145
    #11 0x78ed09 in TmThreadsSlotProcessPkt /root/suricata-4.1.3/src/tm-threads.h:147
    #12 0x78ed09 in AFPReadFromRing /root/suricata-4.1.3/src/source-af-packet.c:1016
    #13 0x7982db in ReceiveAFPLoop /root/suricata-4.1.3/src/source-af-packet.c:1579
    #14 0x804b93 in TmThreadsSlotPktAcqLoop /root/suricata-4.1.3/src/tm-threads.c:348
    #15 0x7f47988dddd4 in start_thread (/lib64/libpthread.so.0+0x7dd4)

Thread T1 (W#01-ens5f1) created by T0 (Suricata-Main) here:
    #0 0x7f4799d31a7f in pthread_create (/lib64/libasan.so.4+0x37a7f)
    #1 0x8085d5 in TmThreadSpawn /root/suricata-4.1.3/src/tm-threads.c:1895
    #2 0x8d1416 in RunModeSetLiveCaptureWorkersForDevice /root/suricata-4.1.3/src/util-runmodes.c:362
    #3 0x8d5569 in RunModeSetLiveCaptureWorkers /root/suricata-4.1.3/src/util-runmodes.c:394
    #4 0x771451 in RunModeIdsAFPWorkers /root/suricata-4.1.3/src/runmode-af-packet.c:845
    #5 0x7835c0 in RunModeDispatch /root/suricata-4.1.3/src/runmodes.c:384
    #6 0x435aa9 in main /root/suricata-4.1.3/src/suricata.c:3003
    #7 0x7f4797c7d3d4 in __libc_start_main (/lib64/libc.so.6+0x223d4)
SUMMARY: AddressSanitizer: heap-buffer-overflow /root/suricata-4.1.3/src/util-streaming-buffer.c:875 in StreamingBufferGetDataAtOffset
Shadow bytes around the buggy address:
  0x0c5a80ef8020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c5a80ef8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c5a80ef8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c5a80ef8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c5a80ef8060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c5a80ef8070: fa fa fa fa fa fa fa fa fa fa fa[fa]fa fa fa fa
  0x0c5a80ef8080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80ef8090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80ef80a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80ef80b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80ef80c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==31185==ABORTING
Actions #15

Updated by Andy Wick almost 4 years ago

This is probably index being -1 I would guess.

Actions #16

Updated by Jackie Cao almost 4 years ago

I also encountered this bug, but 4.1.3 will not crash without IDS rules.

Actions #17

Updated by Victor Julien almost 4 years ago

Hi Andy, are you able to test the following? https://github.com/OISF/suricata/pull/3757 It is a collection of fixes backported from the master branch, and it replaces the whole buffer logic by something much simpler.

Jacky, are you able to test this PR as well? The crash is triggered in http_client_body inspection, which depends on rules being present.

Actions #18

Updated by Andy Wick almost 4 years ago

Ran on 1 box all weekend with no cores

Actions #19

Updated by Victor Julien almost 4 years ago

That is excellent news. Are you going to expand your tests further? Would love to get a bit more confidence this fixes the issue. Thanks!

Actions #20

Updated by Jackie Cao almost 4 years ago

Hi Victor, I found that there are 3 rules that can cause crash.
After I blocked the rules that triggered the crash, I ran for several days without crash.
I will test this PR

Actions #21

Updated by Andy Wick almost 4 years ago

Ran on several more hosts overnight with no cores. All of these hosts were constantly coring before with 4.1.3

Actions #22

Updated by Victor Julien almost 4 years ago

Thanks Andy, that is good news.

Jacky, are you able to share the rule (id's) for the offending 3 rules?

Actions #23

Updated by Victor Julien almost 4 years ago

  • Status changed from New to Assigned
  • Assignee set to Victor Julien
  • Target version set to 4.1.4
Actions #24

Updated by Victor Julien almost 4 years ago

  • Status changed from Assigned to Closed
Actions #25

Updated by Victor Julien almost 4 years ago

  • Related to Bug #2936: Several crashes past week Suricata 4.1.3 , last : double free or corruption added
Actions

Also available in: Atom PDF