Actions
Bug #3436
opensuricatasc: crashing using command 'reopen-log-files'
Affected Versions:
Effort:
Difficulty:
medium
Label:
Description
Creating per Victor's request.
Suricata is core dumping and seg faulting when calling the socket control command "reopen-log-files" in an aggressive manner. Tested on Suricata 5.0.1.
This may be a race condition between the output handlers doing their "reopen" action after receiving a signal and the pcap processing thread still using (or reusing) those same files. From a security point of view, if a bad actor can already issue socket control commands, he can shutdown Suricata if he wants (with the "shutdown" command) and doesn't have to go about crashing it if he want to DoS it.
See attached sctest.py file (tested with Python3.7) for a script that reproduces the issue. (Adjust accordingly.) The pcap and ruleset shouldn't really matter and the config is what ships with Suricata except it has "unix-command" configured like this:
unix-command: enabled: yes filename: test.sock
Suricata is run like this:
-c /src/suricata-current/suricata.yaml -k none --runmode single --unix-socket=test.sock
Some GDB output from crashes:
[Thread 0x7ffff3f32700 (LWP 1696) exited]
free(): invalid pointer
Thread 2 "US" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff4733700 (LWP 1639)]
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x00007ffff5adf801 in __GI_abort () at abort.c:79
#2 0x00007ffff5b28897 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff5c55b9a "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#3 0x00007ffff5b2f90a in malloc_printerr (str=str@entry=0x7ffff5c53d88 "free(): invalid pointer") at malloc.c:5350
#4 0x00007ffff5b36e1c in _int_free (have_lock=0, p=0x7fffe51bb3c0, av=0x7fffe4000020) at malloc.c:4157
#5 __GI___libc_free (mem=0x7fffe51bb3d0) at malloc.c:3124
#6 0x00007ffff697cfe5 in json_delete () from /usr/lib/x86_64-linux-gnu/libjansson.so.4
#7 0x00007ffff697741e in ?? () from /usr/lib/x86_64-linux-gnu/libjansson.so.4
#8 0x00007ffff69774c9 in ?? () from /usr/lib/x86_64-linux-gnu/libjansson.so.4
#9 0x00007ffff697cf91 in json_delete () from /usr/lib/x86_64-linux-gnu/libjansson.so.4
#10 0x00007ffff697741e in ?? () from /usr/lib/x86_64-linux-gnu/libjansson.so.4
#11 0x00007ffff69774c9 in ?? () from /usr/lib/x86_64-linux-gnu/libjansson.so.4
#12 0x00007ffff697cf91 in json_delete () from /usr/lib/x86_64-linux-gnu/libjansson.so.4
#13 0x0000555555736e7b in json_decref (json=0x7fffdc785880) at /usr/include/jansson.h:129
#14 UnixCommandExecute (this=0x555555c987a0 <command>, client=0x7fffe4d8ce20, command=0x7ffff4730ad0 "{\"command\": \"command-list\"}") at unix-manager.c:531
#15 UnixCommandRun (client=client@entry=0x7fffe4d8ce20, this=0x555555c987a0 <command>) at unix-manager.c:622
#16 0x00005555557380f8 in UnixMain (this=0x555555c987a0 <command>) at unix-manager.c:673
#17 UnixManager (th_v=0x555556e491e0, thread_data=<optimized out>) at unix-manager.c:1122
#18 0x0000555555730c9e in TmThreadsManagement (td=0x555556e491e0) at tm-threads.c:722
#19 0x00007ffff675c6db in start_thread (arg=0x7ffff4733700) at pthread_create.c:463
#20 0x00007ffff5bc088f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) continue
Continuing.
[Thread 0x7ffff7feb680 (LWP 1635) exited]
Program terminated with signal SIGABRT, Aborted.
The program no longer exists.
[Thread 0x7ffff3f32700 (LWP 1744) exited] Thread 2 "US" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff4733700 (LWP 1702)] __memset_sse2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:200 200 ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S: No such file or directory. (gdb) bt #0 __memset_sse2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:200 #1 0x00007ffff5b393a2 in __libc_calloc (n=<optimized out>, elem_size=<optimized out>) at malloc.c:3488 #2 0x0000555555585c30 in AlertFastLogInitCtx (conf=0x555555eea290) at alert-fastlog.c:240 #3 0x0000555555704221 in RunModeInitializeOutputs () at runmodes.c:796 #4 0x000055555572c6b3 in PreRunPostPrivsDropInit (runmode=runmode@entry=2) at suricata.c:2322 #5 0x00005555556fec6c in UnixSocketPcapFilesCheck (data=<optimized out>) at runmode-unix-socket.c:575 #6 0x0000555555737f26 in UnixCommandBackgroundTasks (this=0x555555c987a0 <command>) at unix-manager.c:448 #7 UnixManager (th_v=0x555556e48f60, thread_data=<optimized out>) at unix-manager.c:1138 #8 0x0000555555730c9e in TmThreadsManagement (td=0x555556e48f60) at tm-threads.c:722 #9 0x00007ffff675c6db in start_thread (arg=0x7ffff4733700) at pthread_create.c:463 #10 0x00007ffff5bc088f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 (gdb) continue Continuing. [Thread 0x7ffff7feb680 (LWP 1701) exited] Program terminated with signal SIGSEGV, Segmentation fault. The program no longer exists.
[Thread 0x7ffff3f32700 (LWP 1949) exited]
Thread 2 "US" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff4733700 (LWP 1867)]
_int_malloc (av=av@entry=0x7fffe4000020, bytes=bytes@entry=24) at malloc.c:3647
3647 malloc.c: No such file or directory.
(gdb) bt
#0 _int_malloc (av=av@entry=0x7fffe4000020, bytes=bytes@entry=24) at malloc.c:3647
#1 0x00007ffff5b390b1 in __libc_calloc (n=<optimized out>, elem_size=<optimized out>) at malloc.c:3436
#2 0x00005555556c42bd in OutputRegisterFileRotationFlag (flag=flag@entry=0x7fffe51fa2c8) at output.c:873
#3 0x000055555575f9b8 in SCConfLogOpenGeneric (conf=conf@entry=0x555555eea290, log_ctx=log_ctx@entry=0x7fffe51fa210, default_filename=default_filename@entry=0x5555559681df "fast.log", rotate=rotate@entry=1)
at util-logopenfile.c:441
#4 0x0000555555585c1d in AlertFastLogInitCtx (conf=0x555555eea290) at alert-fastlog.c:235
#5 0x0000555555704221 in RunModeInitializeOutputs () at runmodes.c:796
#6 0x000055555572c6b3 in PreRunPostPrivsDropInit (runmode=runmode@entry=2) at suricata.c:2322
#7 0x00005555556fec6c in UnixSocketPcapFilesCheck (data=<optimized out>) at runmode-unix-socket.c:575
#8 0x0000555555737f26 in UnixCommandBackgroundTasks (this=0x555555c987a0 <command>) at unix-manager.c:448
#9 UnixManager (th_v=0x555556e491e0, thread_data=<optimized out>) at unix-manager.c:1138
#10 0x0000555555730c9e in TmThreadsManagement (td=0x555556e491e0) at tm-threads.c:722
#11 0x00007ffff675c6db in start_thread (arg=0x7ffff4733700) at pthread_create.c:463
#12 0x00007ffff5bc088f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) continue
Continuing.
[Thread 0x7ffff7feb680 (LWP 1866) exited]
Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.
[Thread 0x7ffff3f32700 (LWP 2025) exited]
Thread 2 "US" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff4733700 (LWP 1958)]
0x00007ffff6977367 in ?? () from /usr/lib/x86_64-linux-gnu/libjansson.so.4
(gdb) bt
#0 0x00007ffff6977367 in ?? () from /usr/lib/x86_64-linux-gnu/libjansson.so.4
#1 0x00007ffff6977668 in ?? () from /usr/lib/x86_64-linux-gnu/libjansson.so.4
#2 0x00007ffff697d219 in json_object_set_new_nocheck () from /usr/lib/x86_64-linux-gnu/libjansson.so.4
#3 0x00005555557355b7 in UnixManagerReopenLogFiles (cmd=<optimized out>, server_msg=0x7fffe55bfca0, data=<optimized out>) at unix-manager.c:914
#4 0x0000555555736e06 in UnixCommandExecute (this=0x555555c987a0 <command>, client=0x7fffe5964af0, command=0x7ffff4730ad0 "{\"command\": \"reopen-log-files\"}") at unix-manager.c:504
#5 UnixCommandRun (client=client@entry=0x7fffe5964af0, this=0x555555c987a0 <command>) at unix-manager.c:622
#6 0x00005555557380f8 in UnixMain (this=0x555555c987a0 <command>) at unix-manager.c:673
#7 UnixManager (th_v=0x555556e491e0, thread_data=<optimized out>) at unix-manager.c:1122
#8 0x0000555555730c9e in TmThreadsManagement (td=0x555556e491e0) at tm-threads.c:722
#9 0x00007ffff675c6db in start_thread (arg=0x7ffff4733700) at pthread_create.c:463
#10 0x00007ffff5bc088f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) continue
Continuing.
[Thread 0x7ffff7feb680 (LWP 1956) exited]
Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.
Files
Updated by Philippe Antoine almost 2 years ago
- Assignee set to OISF Dev
- Target version set to 8.0.0-beta1
Updated by Victor Julien about 2 months ago
- Status changed from New to Assigned
- Assignee changed from OISF Dev to Jeff Lucovsky
- Target version changed from 8.0.0-beta1 to 8.0.0-rc1
Updated by Jeff Lucovsky about 1 month ago
- Status changed from Assigned to In Progress
Updated by Shivani Bhardwaj 6 days ago
- Subject changed from Suricata Socket Control crashing using command 'reopen-log-files' to suricatasc: crashing using command 'reopen-log-files'
Updated by Jeff Lucovsky 3 days ago ยท Edited
Updated command line: -c /src/suricata-current/suricata.yaml -k none --runmode single --unix-socket=test.sock --set security.limit-noproc=false
Updated by Jeff Lucovsky 3 days ago
Stack (Suricata 8 beta1)
malloc(): unsorted double linked list corrupted
Thread 2 "US" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff46006c0 (LWP 3232242)]
__pthread_kill_implementation (threadid=<optimized out>, signo=6, no_tid=0) at ./nptl/pthread_kill.c:44
warning: 44 ./nptl/pthread_kill.c: No such file or directory
(gdb) bt
#0 __pthread_kill_implementation (threadid=<optimized out>, signo=6, no_tid=0) at ./nptl/pthread_kill.c:44
#1 __pthread_kill_internal (threadid=<optimized out>, signo=6) at ./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3 0x00007ffff6e4519e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4 0x00007ffff6e28902 in __GI_abort () at ./stdlib/abort.c:79
#5 0x00007ffff6e2976c in __libc_message_impl (fmt=fmt@entry=0x7ffff6fdc8e2 "%s\n") at ../sysdeps/posix/libc_fatal.c:134
#6 0x00007ffff6eae565 in malloc_printerr (str=str@entry=0x7ffff6fdfe98 "malloc(): unsorted double linked list corrupted") at ./malloc/malloc.c:5772
#7 0x00007ffff6eb146c in _int_malloc (av=av@entry=0x7fffec000030, bytes=bytes@entry=64) at ./malloc/malloc.c:4086
#8 0x00007ffff6eb3ebe in __libc_calloc (n=n@entry=1, elem_size=elem_size@entry=64) at ./malloc/malloc.c:3754
#9 0x00005555556c8ea7 in SCCallocFunc (nm=3232234, nm@entry=1, sz=3232242, sz@entry=64) at util-mem.c:60
#10 0x0000555555906f6a in UnixListAddFile (this=this@entry=0x55557a3eb8e0, filename=filename@entry=0x7fffed4019d0 "/home/jlucovsky/pcap/55951.pcap",
output_dir=0x6 <error: Cannot access memory at address 0x6>, output_dir@entry=0x7fffd426cf90 "/tmp/sraw2", tenant_id=tenant_id@entry=0, continuous=false,
should_delete=false, delay=30, poll_interval=5) at runmode-unix-socket.c:250
#11 0x0000555555906e75 in UnixSocketAddPcapFileImpl (cmd=<optimized out>, answer=0x7fffec002b50, data=0x55557a3eb8e0, continuous=false)
at runmode-unix-socket.c:379
#12 0x00005555556b0566 in UnixCommandExecute (
command=0x7ffff45fddc0 "{\"command\": \"pcap-file\", \"arguments\": {\"filename\": \"/home/jlucovsky/pcap/55951.pcap\", \"output-dir\": \"/tmp/sraw2\"}}", client=0x7fffec0020a0, this=<optimized out>) at unix-manager.c:499
#13 UnixCommandRun (client=0x7fffec0020a0, this=<optimized out>) at unix-manager.c:615
#14 UnixMain (this=<optimized out>) at unix-manager.c:666
#15 UnixManager (th_v=0x55555700dc60, thread_data=<optimized out>) at unix-manager.c:1161
#16 0x00005555556ae045 in TmThreadsManagement (td=0x55555700dc60) at tm-threads.c:592
#17 0x00007ffff6ea1e2e in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#18 0x00007ffff6f33a4c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
Updated by Jeff Lucovsky 3 days ago
With ASAN:
==3246153==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x5120020fd5b0 at pc 0x555555e75c9d bp 0x7fffedbfec50 sp 0x7fffedbfec48
WRITE of size 4 at 0x5120020fd5b0 thread T1 (US)
[Detaching after fork from child process 3248095]
#0 0x555555e75c9c in OutputNotifyFileRotation /home/jlucovsky/src/jal/3436/src/output.c:715:23
#1 0x5555559853ba in UnixManagerReopenLogFiles /home/jlucovsky/src/jal/3436/src/unix-manager.c:938:5
#2 0x55555598699f in UnixCommandExecute /home/jlucovsky/src/jal/3436/src/unix-manager.c:499:20
#3 0x55555598699f in UnixCommandRun /home/jlucovsky/src/jal/3436/src/unix-manager.c:615:5
#4 0x55555598699f in UnixMain /home/jlucovsky/src/jal/3436/src/unix-manager.c:666:13
#5 0x55555598699f in UnixManager /home/jlucovsky/src/jal/3436/src/unix-manager.c:1161:15
#6 0x5555559827f8 in TmThreadsManagement /home/jlucovsky/src/jal/3436/src/tm-threads.c:592:9
#7 0x555555929dea in asan_thread_start(void*) asan_interceptors.cpp.o
#8 0x7ffff6ea1e2d in start_thread nptl/pthread_create.c:447:8
#9 0x7ffff6f33a4b in __GI___clone3 misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
0x5120020fd5b0 is located 344 bytes after 280-byte region [0x5120020fd340,0x5120020fd458)
allocated by thread T0 (Suricata-Main) here:
#0 0x55555592c67d in calloc (/home/jlucovsky/src/jal/3436/src/suricata+0x3d867d) (BuildId: d1b763ac416557edb24863e005e6025562fce564)
#1 0x5555559b72e6 in SCCallocFunc /home/jlucovsky/src/jal/3436/src/util-mem.c:60:20
#2 0x555555dc120a in SigAlloc /home/jlucovsky/src/jal/3436/src/detect-parse.c:1899:22
#3 0x555555dcf46e in SigInitHelper /home/jlucovsky/src/jal/3436/src/detect-parse.c:2735:22
#4 0x555555dc4672 in SigInitDo /home/jlucovsky/src/jal/3436/src/detect-parse.c:2931:22
#5 0x555555dc5cbb in SigInit /home/jlucovsky/src/jal/3436/src/detect-parse.c:2974:12
#6 0x555555dc5cbb in DetectEngineAppendSig /home/jlucovsky/src/jal/3436/src/detect-parse.c:3316:22
#7 0x555555c102ff in DetectLoadSigFile /home/jlucovsky/src/jal/3436/src/detect-engine-loader.c:176:19
#8 0x555555c0e047 in ProcessSigFiles /home/jlucovsky/src/jal/3436/src/detect-engine-loader.c:270:13
#9 0x555555c0d502 in SigLoadSignatures /home/jlucovsky/src/jal/3436/src/detect-engine-loader.c:418:27
#10 0x555555972c5a in LoadSignatures /home/jlucovsky/src/jal/3436/src/suricata.c:2427:9
#11 0x555555972c5a in PostConfLoadedDetectSetup /home/jlucovsky/src/jal/3436/src/suricata.c:2578:17
#12 0x555555974d01 in SuricataInit /home/jlucovsky/src/jal/3436/src/suricata.c:2981:5
#13 0x55555596dc0f in main /home/jlucovsky/src/jal/3436/src/main.c:54:5
#14 0x7ffff6e2a3b7 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#15 0x7ffff6e2a47a in __libc_start_main csu/../csu/libc-start.c:360:3
#16 0x55555588c664 in _start (/home/jlucovsky/src/jal/3436/src/suricata+0x338664) (BuildId: d1b763ac416557edb24863e005e6025562fce564)
Thread T1 (US) created by T0 (Suricata-Main) here:
#0 0x5555559119a5 in pthread_create (/home/jlucovsky/src/jal/3436/src/suricata+0x3bd9a5) (BuildId: d1b763ac416557edb24863e005e6025562fce564)
#1 0x55555597dab1 in TmThreadSpawn /home/jlucovsky/src/jal/3436/src/tm-threads.c:1717:14
#2 0x555555985471 in UnixManagerThreadSpawn /home/jlucovsky/src/jal/3436/src/unix-manager.c:1200:9
#3 0x555555eca243 in RunModeUnixSocketMaster /home/jlucovsky/src/jal/3436/src/runmode-unix-socket.c:1710:5
#4 0x555555ec6ae1 in RunModeDispatch /home/jlucovsky/src/jal/3436/src/runmodes.c:432:5
#5 0x555555974d7b in SuricataInit /home/jlucovsky/src/jal/3436/src/suricata.c:2999:5
#6 0x55555596dc0f in main /home/jlucovsky/src/jal/3436/src/main.c:54:5
#7 0x7ffff6e2a3b7 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#8 0x7ffff6e2a47a in __libc_start_main csu/../csu/libc-start.c:360:3
#9 0x55555588c664 in _start (/home/jlucovsky/src/jal/3436/src/suricata+0x338664) (BuildId: d1b763ac416557edb24863e005e6025562fce564)
SUMMARY: AddressSanitizer: heap-buffer-overflow /home/jlucovsky/src/jal/3436/src/output.c:715:23 in OutputNotifyFileRotation
Shadow bytes around the buggy address:
0x5120020fd300: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
0x5120020fd380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x5120020fd400: 00 00 00 00 00 00 00 00 00 00 00 fa fa fa fa fa
0x5120020fd480: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x5120020fd500: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x5120020fd580: fa fa fa fa fa fa[fa]fa fa fa fa fa fa fa fa fa
0x5120020fd600: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x5120020fd680: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x5120020fd700: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x5120020fd780: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x5120020fd800: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==3246153==ABORTING
[Thread 0x7fffedc006c0 (LWP 3246217) exited]
[Inferior 1 (process 3246153) exited with code 01]
Updated by Jeff Lucovsky about 23 hours ago
- Status changed from In Progress to In Review
Actions