Project

General

Profile

Actions

Bug #3380

closed

Segfault when using multi-detect

Added by Gatewatcher Dev Team almost 5 years ago. Updated over 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

Hi everyone,

We recently stumbled on a reproducible segmentation fault in Suricata, when
activating multi-tenancy.

The circumstances of this happening seems to be quite rare, but we managed to
reduce the problem and draw a perimeter around them:

  • multi-tenancy must be enabled, with at least one non-null tenant
  • a rule must match on PCRE
  • some traffic must contain fragments to reassemble.

The segmentation fault is caused by the dereferencing of a NULL pointer,
specifically de_ctx->sig_array which is not initialized (the whole structure is
a DETECT_ENGINE_TYPE_MT_STUB ).

The segfault happens during the analysis of a packet created by reassembling
fragmented traffic. The reason why a stub is used here might be that the
tenant_id is unset and the livedev pointer is a null pointer too. In our test
config attached, there is only one interface configured for simplicity's sake,
and no interface in the tenant 0, but the bug happened also when there was
multiple tenants, each with different rulesets, including interfaces associated
to the default tenant (0).

(gdb) bt
#0  0x00000002572f2945 in DetectPrefilterMergeSort (de_ctx=0x26d5fd7b0, det_ctx=0x3a820a59a00) at detect.c:350
#1  0x00000002572f50e5 in DetectRunPrefilterPkt (tv=0x26d603840, de_ctx=0x26d5fd7b0, det_ctx=0x3a820a59a00, p=0x3a83c99f580, scratch=0x3a8b8f1df30) at detect.c:736
#2  0x00000002572f147c in DetectRun (th_v=0x26d603840, de_ctx=0x26d5fd7b0, det_ctx=0x3a820a59a00, p=0x3a83c99f580) at detect.c:131
#3  0x00000002572fa53e in DetectFlow (tv=0x26d603840, de_ctx=0x26d5fd7b0, det_ctx=0x3a820a59a00, p=0x3a83c99f580) at detect.c:1661
#4  0x00000002572faa60 in Detect (tv=0x26d603840, p=0x3a83c99f580, data=0x3a820a59a00, pq=0x0, postpq=0x0) at detect.c:1735
#5  0x000000025743c53a in FlowWorker (tv=0x26d603840, p=0x3a83c99f580, data=0x3a83c9b6170, preq=0x26d603c20, unused=0x0) at flow-worker.c:260
#6  0x00000002575a039e in TmThreadsSlotVarRun (tv=0x26d603840, p=0x3a83c99f580, slot=0x26d603be0) at tm-threads.c:145
#7  0x00000002575a0470 in TmThreadsSlotVarRun (tv=0x26d603840, p=0x3a83c99ff20, slot=0x26d603aa0) at tm-threads.c:171
#8  0x00000002574ed291 in TmThreadsSlotProcessPkt (tv=0x26d603840, s=0x26d603aa0, p=0x3a83c99ff20) at tm-threads.h:147
#9  0x00000002574f22a5 in AFPParsePacketV3 (ptv=0x3a83c9a08c0, pbd=0x3a126400000, ppd=0x3a1264b8f18) at source-af-packet.c:1124
#10 0x00000002574f258d in AFPWalkBlock (ptv=0x3a83c9a08c0, pbd=0x3a126400000) at source-af-packet.c:1140
#11 0x00000002574f2bd0 in AFPReadFromRingV3 (ptv=0x3a83c9a08c0) at source-af-packet.c:1190
#12 0x00000002574f4d32 in ReceiveAFPLoop (tv=0x26d603840, data=0x3a83c9a08c0, slot=0x26d603960) at source-af-packet.c:1583
#13 0x00000002575a14e2 in TmThreadsSlotPktAcqLoop (td=0x26d603840) at tm-threads.c:348
#14 0x000003acae793a43 in start_thread () from /lib64/libpthread.so.0
#15 0x000003acacae990f in clone () from /lib64/libc.so.6
(gdb) p de_ctx->type
$1 = DETECT_ENGINE_TYPE_MT_STUB
(gdb) p de_ctx->sig_array
$2 = (Signature **) 0x0
(gdb) f 1
#1  0x00000002572f50e5 in DetectRunPrefilterPkt (tv=0x26d603840, de_ctx=0x26d5fd7b0, det_ctx=0x3a820a59a00, p=0x3a83c99f580, scratch=0x3a8b8f1df30) at detect.c:736
736    in detect.c
(gdb) p p->flags
$3 = 37757184
(gdb) p p->tenant_id
$4 = 0
(gdb) p p->livedev
$5 = (struct LiveDevice_ *) 0x0

Another thing that we noticed and that we cannot fully explain is that the
first packet of the attached pcap seems to be necessary for the segfault to
occur. When removed, we could not reproduce the issue. It might be that this
packet "starts" the block and the other two are stored in the slots as "extra
packets"?

Please reach out for any additional details or tests.

Thank you.

Cheers,
Florian Maury for ArmatureTech Dev Team


Files

crash.rules (82 Bytes) crash.rules Ruleset containing the rule that triggers the crash Gatewatcher Dev Team, 12/05/2019 01:41 PM
crash_zero.pcap (3.41 KB) crash_zero.pcap A sample pcap containing packets that triggers the crash Gatewatcher Dev Team, 12/05/2019 01:41 PM
multi-detect-eno1.yaml (213 Bytes) multi-detect-eno1.yaml Basic config for multi tenancy Gatewatcher Dev Team, 12/05/2019 01:41 PM
suricata.yaml (5.72 KB) suricata.yaml Basic config using multi-tenancy Gatewatcher Dev Team, 12/05/2019 01:41 PM
Actions #1

Updated by Gatewatcher Dev Team almost 5 years ago

Just a word of context, the coredump hereabove was generated by Suricata 4.1.5. The attached config files were on Suricata 5.0 on which we also reproduced the bug.

Actions #2

Updated by Victor Julien almost 5 years ago

  • Status changed from New to Assigned
  • Assignee set to Jason Ish
Actions #3

Updated by Gatewatcher Dev Team almost 5 years ago

Hi Jason, Victor,

Did you manage to reproduce the issue in your lab with the attached files?

This issue is a blocking one for us and our customers needing multi-tenancy. It also constitutes an easy network-triggered DoS for any Suricata user activating multi-tenancy.

At any rate, if you could provide any advice, workaround for the issue or direction regarding an acceptable solution, these would be well met.

Thank you.

Cheers,
Florian Maury for ArmatureTech Dev Team

Actions #4

Updated by Victor Julien over 4 years ago

  • Target version set to 5.0.2
Actions #5

Updated by Victor Julien over 4 years ago

  • Status changed from Assigned to Closed
  • Affected Versions 5.0.1 added
  • Affected Versions deleted (4.1.5)
Actions

Also available in: Atom PDF