Bug #6782
openCrasher in HTTP chunked / StreamingBuffer
Description
Seeing similar crashes on multiple sites, looks like a heap corruption somewhere:
SIGSEGV Thread 2 (Thread 0x7ffff5d5c6c0 (LWP 130) "W#01-eno1"): #0 0x00007ffff7990507 in __memmove_avx_unaligned_erms_rtm () from /lib64/libc.so.6 #1 0x000055555587a2eb in StreamingBufferAppend () #2 0x000055555588d75a in HtpBodyAppendChunk () #3 0x0000555555792d28 in HTPCallbackRequestBodyData () #4 0x00007ffff7fa3779 in htp_hook_run_all () from /lib64/libhtp.so.2 #5 0x00007ffff7fb20eb in htp_req_run_hook_body_data () from /lib64/libhtp.so.2 #6 0x00007ffff7fac63d in htp_tx_req_process_body_data_ex () from /lib64/libhtp.so.2 #7 0x00007ffff7facacf in htp_connp_REQ_BODY_IDENTITY () from /lib64/libhtp.so.2 #8 0x00007ffff7fa844d in htp_connp_req_data () from /lib64/libhtp.so.2 #9 0x000055555578f7d2 in HTPHandleRequestData () #10 0x0000555555799066 in AppLayerParserParse () #11 0x0000555555786b0c in AppLayerHandleTCPData () #12 0x0000555555862335 in StreamTcpReassembleAppLayer () #13 0x0000555555863004 in StreamTcpReassembleHandleSegment () #14 0x00005555558510e6 in StreamTcpPacketStateEstablished.lto_priv.0 () #15 0x00005555558574b8 in StreamTcpStateDispatch () #16 0x000055555585a439 in StreamTcpPacket () #17 0x000055555581b9bf in FlowWorkerStreamTCPUpdate () #18 0x000055555581bd20 in FlowWorker.lto_priv.0 () #19 0x000055555576dab3 in TmThreadsSlotVarRun () #20 0x00005555558473af in AFPReadFromRingV3 () #21 0x0000555555848de4 in ReceiveAFPLoop.lto_priv.0 () #22 0x000055555576fd74 in TmThreadsSlotPktAcqLoop () #23 0x00007ffff78b7897 in start_thread () from /lib64/libc.so.6 #24 0x00007ffff793e674 in clone () from /lib64/libc.so.6
SIGSEGV Thread 2 (Thread 0x7ffff5d5b6c0 (LWP 104) "W#01-eno1"): #0 0x00007ffff798f3ed in __memmove_avx_unaligned_erms_rtm () from /lib64/libc.so.6 #1 0x00005555557dd712 in StreamingBufferAppendNoTrack () #2 0x00005555557c9c40 in AppendData () #3 0x00005555557ca2bb in FileOpenFile.lto_priv.0 () #4 0x00005555556f856c in HTPFileOpen () #5 0x00005555556f262e in HTPCallbackRequestBodyData () #6 0x00007ffff7fa2899 in htp_hook_run_all () from /lib64/libhtp.so.2 #7 0x00007ffff7fb1d0b in htp_req_run_hook_body_data () from /lib64/libhtp.so.2 #8 0x00007ffff7fab97d in htp_tx_req_process_body_data_ex () from /lib64/libhtp.so.2 #9 0x00007ffff7fabe0f in htp_connp_REQ_BODY_IDENTITY () from /lib64/libhtp.so.2 #10 0x00007ffff7fa73ed in htp_connp_req_data () from /lib64/libhtp.so.2 #11 0x00005555556ed082 in HTPHandleRequestData () #12 0x00005555556fa7d6 in AppLayerParserParse () #13 0x00005555556efcdc in AppLayerHandleTCPData () #14 0x00005555557c53c5 in StreamTcpReassembleAppLayer () #15 0x00005555557c6094 in StreamTcpReassembleHandleSegment () #16 0x00005555557b4176 in StreamTcpPacketStateEstablished.lto_priv.0 () #17 0x00005555557ba548 in StreamTcpStateDispatch () #18 0x00005555557bd4c9 in StreamTcpPacket () #19 0x000055555577d40f in FlowWorkerStreamTCPUpdate () #20 0x000055555577d770 in FlowWorker.lto_priv.0 () #21 0x00005555556ce998 in TmThreadsSlotVarRun () #22 0x00005555557aa9bf in AFPReadFromRingV3 () #23 0x00005555557ac414 in ReceiveAFPLoop.lto_priv.0 () #24 0x00005555556d0cc4 in TmThreadsSlotPktAcqLoop () #25 0x00007ffff78b6897 in start_thread () from /lib64/libc.so.6 #26 0x00007ffff793d674 in clone () from /lib64/libc.so.6
SIGABRT Thread 2 (Thread 0x7ffff5d5b6c0 (LWP 104) "W#01-eno1"): #0 0x00007ffff78b8834 in __pthread_kill_implementation () from /lib64/libc.so.6 #1 0x00007ffff78668ee in raise () from /lib64/libc.so.6 #2 0x00007ffff784e8ff in abort () from /lib64/libc.so.6 #3 0x00007ffff784f7d0 in __libc_message.cold () from /lib64/libc.so.6 #4 0x00007ffff78c27a5 in malloc_printerr () from /lib64/libc.so.6 #5 0x00007ffff78c5abc in _int_malloc () from /lib64/libc.so.6 #6 0x00007ffff78c7dae in calloc () from /lib64/libc.so.6 #7 0x00005555557b389c in StreamTcpPacketStateSynSent.lto_priv.0 () #8 0x00005555557ba4a0 in StreamTcpStateDispatch () #9 0x00005555557bd4c9 in StreamTcpPacket () #10 0x000055555577d40f in FlowWorkerStreamTCPUpdate () #11 0x000055555577d770 in FlowWorker.lto_priv.0 () #12 0x00005555556ce998 in TmThreadsSlotVarRun () #13 0x00005555557aa9bf in AFPReadFromRingV3 () #14 0x00005555557ac414 in ReceiveAFPLoop.lto_priv.0 () #15 0x00005555556d0cc4 in TmThreadsSlotPktAcqLoop () #16 0x00007ffff78b6897 in start_thread () from /lib64/libc.so.6 #17 0x00007ffff793d674 in clone () from /lib64/libc.so.6
Files
Updated by Victor Julien 2 months ago
Do you have a reproducer? Which libhtp version do you see this with? Do you see this only with libhtp 0.5.46?
Updated by Gianni Tedesco 2 months ago
Ran with:
/usr/sbin/suricata -c suricata.conf --init-errors-fatal -k none -l /var/run --af-packet=eno1
Nothing in the log except the usual startup guff. Working on getting the config.
We don't have a reproducer, our best guess at the moment is to maybe get a pcap of HTTP traffic, but not sure if that's the way to go? It involves reaching out to affected customers and asking them to do it, so we can't directly get that.
The version of libhtp we built with is just whatever shipped with the release, we're building from release tags on github, first time we saw this was with 7.0.1, and we've just upgraded to 7.0.3 to repro it.
Updated by Victor Julien 2 months ago
Ok, thank you. That means it's not a new issue with libhtp 0.5.46.
Updated by Gianni Tedesco about 1 month ago
A bit of extra context here. The systems this is happening on, it's happening pretty regularly (eg. every 10 minutes), the issue is that they're on 10GB NICs, which are almost fully saturated with traffic, single threaded (due to broadcom NICs having no symmetric RSS), and the drop rate for suri is often around 10-15% so i expect there's a lot of missing packets in the stream and stuff like that. Not sure if that's helpful or not.
Updated by Victor Julien about 1 month ago
I think it would be useful to do a symbols enabled build and get a full bt, or even an ASAN enabled build.
Updated by Gianni Tedesco 24 days ago
- File StreamingBufferAppend_crash.txt StreamingBufferAppend_crash.txt added
- File StreamingBufferInit_crash.txt StreamingBufferInit_crash.txt added
Ran with ASAN and debug compile and got the following output, not sure much more helpful it is than previous backtrace:
-Og -fno-inline-functions -fno-lto