Security #8021
closedeve/alert: heap buffer overflow on verdict
Description
While running suricata 8.0.1 we have been getting crashes related to memory issues. The system is operating fine for multiple days before crashing with a segfault.
Since we weren't able to reproduce the issue with a test system, we enabled sanitzer support on the target server.
This is the asan log from the most recent crash:
==3404==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7cd235b9a1dc at pc 0x564b10d7a6f9 bp 0x7b71feb87a50 sp 0x7b71feb87a48
READ of size 1 at 0x7cd235b9a1dc thread T8 (W#03-eth1)
#0 0x564b10d7a6f8 in EveAddVerdict /src/suricata-8.0.1/src/output-json-alert.c:581:48
#1 0x564b10d7f0a0 in AlertJson /src/suricata-8.0.1/src/output-json-alert.c:795:13
#2 0x564b10d7a958 in JsonAlertLogger /src/suricata-8.0.1/src/output-json-alert.c:874:16
#3 0x564b10db881f in OutputPacketLog /src/suricata-8.0.1/src/output-packet.c:106:13
#4 0x564b1093aa72 in OutputLoggerLog /src/suricata-8.0.1/src/output.c:809:9
#5 0x564b108c8c12 in FlowWorker /src/suricata-8.0.1/src/flow-worker.c:673:5
#6 0x564b1037e977 in TmThreadsSlotVarRun /src/suricata-8.0.1/src/tm-threads.c:137:21
#7 0x564b10972a6a in TmThreadsSlotProcessPkt /src/suricata-8.0.1/src/./tm-threads.h:202:17
#8 0x564b1096d287 in AFPReadFromRing /src/suricata-8.0.1/src/source-af-packet.c:935:13
#9 0x564b10964b09 in ReceiveAFPLoop /src/suricata-8.0.1/src/source-af-packet.c:1421:17
#10 0x564b103a32b4 in TmThreadsSlotPktAcqLoop /src/suricata-8.0.1/src/tm-threads.c:334:13
#11 0x564b10327b0b in asan_thread_start(void*) /src/compiler-rt-21.1.4.src/lib/asan/asan_interceptors.cpp:239:28
#12 0x7f722ad9972b in start_thread pthread_create.c
#13 0x7f722ae02627 in __GI___clone3 (/lib64/libc.so.6+0xef627)
0x7cd235b9a1dc is located 4 bytes after 600-byte region [0x7cd235b99f80,0x7cd235b9a1d8)
allocated by thread T8 (W#03-eth1) here:
#0 0x564b1032ca09 in calloc /src/compiler-rt-21.1.4.src/lib/asan/asan_malloc_linux.cpp:74:3
#1 0x564b103fbc1d in SCCallocFunc /src/suricata-8.0.1/src/util-mem.c:60:20
#2 0x564b109435f8 in PacketInit /src/suricata-8.0.1/src/packet.c:66:24
#3 0x564b10558173 in PacketGetFromAlloc /src/suricata-8.0.1/src/decode.c:264:5
#4 0x564b103af584 in PacketPoolInit /src/suricata-8.0.1/src/tmqh-packetpool.c:254:21
#5 0x564b103a4807 in TmThreadsSlotPktAcqLoopInit /src/suricata-8.0.1/src/tm-threads.c:217:5
#6 0x564b103a31db in TmThreadsSlotPktAcqLoop /src/suricata-8.0.1/src/tm-threads.c:327:10
#7 0x564b10327b0b in asan_thread_start(void*) /src/compiler-rt-21.1.4.src/lib/asan/asan_interceptors.cpp:239:28
Thread T8 (W#03-eth1) created by T0 (Suricata-Main) here:
#0 0x564b1030efc1 in pthread_create /src/compiler-rt-21.1.4.src/lib/asan/asan_interceptors.cpp:250:3
#1 0x564b10394ca1 in TmThreadSpawn /src/suricata-8.0.1/src/tm-threads.c:1745:14
#2 0x564b10e1ba0d in RunModeSetLiveCaptureWorkersForDevice /src/suricata-8.0.1/src/util-runmodes.c:322:13
#3 0x564b10e1b4e6 in RunModeSetLiveCaptureWorkers /src/suricata-8.0.1/src/util-runmodes.c:347:9
#4 0x564b10dbf313 in RunModeIdsAFPWorkers /src/suricata-8.0.1/src/runmode-af-packet.c:877:11
#5 0x564b10956dc2 in RunModeDispatch /src/suricata-8.0.1/src/runmodes.c:442:5
#6 0x564b1037a10f in SuricataInit /src/suricata-8.0.1/src/suricata.c:3091:5
#7 0x564b1036e83d in main /src/suricata-8.0.1/src/main.c:57:5
#8 0x7f722ad3b63e in __libc_start_call_main libc-start.c
#9 0x7f722ad3b6eb in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x286eb)
SUMMARY: AddressSanitizer: heap-buffer-overflow /src/suricata-8.0.1/src/output-json-alert.c:581:48 in EveAddVerdict
Shadow bytes around the buggy address:
0x7cd235b99f00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x7cd235b99f80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x7cd235b9a000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x7cd235b9a080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x7cd235b9a100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x7cd235b9a180: 00 00 00 00 00 00 00 00 00 00 00[fa]fa fa fa fa
0x7cd235b9a200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x7cd235b9a280: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x7cd235b9a300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x7cd235b9a380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x7cd235b9a400: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==3404==ABORTING
Updated by Victor Julien 2 months ago
- Assignee set to OISF Dev
- Priority changed from Normal to High
- Target version changed from TBD to 9.0.0-beta1
- Label Needs backport to 8.0 added
Thanks for your report. Is the crash always in the same function? If other crashed look different, please add them to the ticket as well.
Updated by Juliana Fajardini Reichow 2 months ago
- Affected Versions git main added
Updated by Juliana Fajardini Reichow 2 months ago
- Related to Optimization #5180: detect/alert: make sure that signatures with `drop` action are respected, even if the alert is discarded added
Updated by Juliana Fajardini Reichow 2 months ago
- Assignee changed from OISF Dev to Juliana Fajardini Reichow
Assigning it to me as I investigate it further.
Updated by Philippe Antoine 2 months ago
- Tracker changed from Bug to Security
- Status changed from New to In Review
- Severity set to MODERATE
Gitlab MR
Updated by Juliana Fajardini Reichow 2 months ago
- Assignee changed from Juliana Fajardini Reichow to Philippe Antoine
Updated by Philippe Antoine 2 months ago
- Assignee changed from Philippe Antoine to Juliana Fajardini Reichow
Juliana, I proposed a test and a POC fix, but you look like you know better this code, and will propose better tests and fix, right ?
Updated by Jules Lumbergh 2 months ago
Victor Julien wrote in #note-1:
Thanks for your report. Is the crash always in the same function? If other crashed look different, please add them to the ticket as well.
Hello Victor, thank you for your fast response. This is the first crash where we got an asan report from. If we get a another, different looking asan report, I'll add it to the ticket.
Updated by Juliana Fajardini Reichow 2 months ago
Philippe Antoine wrote in #note-13:
Juliana, I proposed a test and a POC fix, but you look like you know better this code, and will propose better tests and fix, right ?
I can! I had assigned the ticket to you because I saw that you had shared proposed solutions, but I can take it over.
Updated by Philippe Antoine 2 months ago
- Subject changed from Suricata crash - asan heap-buffer-overflow to output/verdict : asan heap-buffer-overflow
Updated by Shivani Bhardwaj 2 months ago
- Subject changed from output/verdict : asan heap-buffer-overflow to output/alert: heap buffer overflow on verdict
Updated by Victor Julien 2 months ago
- Subject changed from output/alert: heap buffer overflow on verdict to eve/alert: heap buffer overflow on verdict
Updated by Juliana Fajardini Reichow 2 months ago
- Related to Bug #7630: eve/alert: incorrect verdict with pass + alert rule added
Updated by Juliana Fajardini Reichow 2 months ago
- Assignee changed from Juliana Fajardini Reichow to Philippe Antoine
Once again assigning to Philippe, as I think his patch fixes it best.
Updated by Philippe Antoine 2 months ago
- Assignee changed from Philippe Antoine to Victor Julien
I am not the one working on the good fix for this ;-p
Updated by Juliana Fajardini Reichow 2 months ago
- CVE set to 2025-64330
Updated by Juliana Fajardini Reichow 28 days ago
- Private changed from Yes to No