https://redmine.openinfosecfoundation.org/https://redmine.openinfosecfoundation.org/favicon.ico?17011170022019-11-19T00:10:31ZOpen Information Security FoundationSuricata - Bug #3342: Suricata 5.0 crashes while parsing SMB datahttps://redmine.openinfosecfoundation.org/issues/3342?journal_id=146882019-11-19T00:10:31ZMichal Purzynski
<ul></ul><p>What I see around this time (with Zeek's flow logs and parsed SMB data) is</p>
<p>- there's a couple of connections between 150 - 500 seconds<br />- two connections over 500 seconds have a very high bytes count - over 3.4GB transferred</p>
<p>Connections are multiplexed - there's a long-running TCP connection with multiple SMB operations over it. I can count at least 144 operations.</p>
<p>Out of these 144 SMB operations per one TCP connection</p>
<p>- most are in the range of tens of KBs to 200KBs<br />- there's at least one that literally transfers entire Windows image - 3669480484 bytes of it<br />"Operating Systems\Windows 10 Enterprise x64\Sources\install.wim"</p>
<p>Zeek has no problem tracing that. My guess is that there's some problem with the memory management in conditions like these.</p> Suricata - Bug #3342: Suricata 5.0 crashes while parsing SMB datahttps://redmine.openinfosecfoundation.org/issues/3342?journal_id=146902019-11-19T06:51:10ZVictor Julienvictor@inliniac.net
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Assigned</i></li><li><strong>Assignee</strong> set to <i>Victor Julien</i></li></ul><p>Thanks Michal. This crash happens during cleanup after the flow has timed out. I strongly suspect the cause of this is elsewhere in the form of some memory corruption. Are you able to rebuild suricata with ASAN enabled and run that for a while?</p> Suricata - Bug #3342: Suricata 5.0 crashes while parsing SMB datahttps://redmine.openinfosecfoundation.org/issues/3342?journal_id=146912019-11-19T06:52:48ZVictor Julienvictor@inliniac.net
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/14691/diff?detail_id=15156">diff</a>)</li></ul> Suricata - Bug #3342: Suricata 5.0 crashes while parsing SMB datahttps://redmine.openinfosecfoundation.org/issues/3342?journal_id=146922019-11-19T06:58:53ZMichal Purzynski
<ul></ul><p>Victor Julien wrote:</p>
<blockquote>
<p>Thanks Michal. This crash happens during cleanup after the flow has timed out. I strongly suspect the cause of this is elsewhere in the form of some memory corruption. Are you able to rebuild suricata with ASAN enabled and run that for a while?</p>
</blockquote>
<p>Here's the crash message, looks like a segfault. Double free?</p>
<p>[1363579.695283] FR#01<sup><a href="#fn187548">187548</a></sup>: segfault at 7fe28bffa008 ip 00007fe629cab745 sp 00007fe5acfe87d0 error 4 cpu 0 in libc-2.30.so[7fe629c2e000+188000]<br />[1363579.695298] Code: 00 0f 1f 40 00 f3 0f 1e fa 48 83 ec 18 48 8b 05 b9 77 15 00 48 8b 00 48 85 c0 0f 85 8d 00 00 00 0f 1f 44 00 00 48 85 ff 74 7b <48> 8b 47 f8 48 8d 77 f0 a8 02 75 3f 2e 2e 2e 48 8b 15 0d 76 15 00</p>
<p>Glibc's sources tell me we're crashing in</p>
<p>void<br />__libc_free (void <strong>mem)<br />(...)<br /> if (chunk_is_mmapped (p)) /</strong> release mmapped memory. */</p>
<p>I'll rebuild Suri with ASAN.</p> Suricata - Bug #3342: Suricata 5.0 crashes while parsing SMB datahttps://redmine.openinfosecfoundation.org/issues/3342?journal_id=147152019-11-20T22:22:23ZMichal Purzynski
<ul></ul><p>After 30+ hours Suricata crashed again, here's what ASAN reported (not much, to be honest)</p>
<pre>
AddressSanitizer:DEADLYSIGNAL
=================================================================
==258009==ERROR: AddressSanitizer: SEGV on unknown address 0x7ff3dfee07f0 (pc 0x7ff813bf8dd6 bp 0x7ff3dfee07f0 sp 0x7ff6d5afed40 T12)
==258009==The signal is caused by a WRITE memory access.
#0 0x7ff813bf8dd5 (/usr/lib64/libasan.so.5+0x2bdd5)
#1 0x7ff813ce207c in free (/usr/lib64/libasan.so.5+0x11507c)
#2 0x55ace165835d in StreamingBufferClear /home/clear/tmp/suricata/src/util-streaming-buffer.c:139
#3 0x55ace1658381 in StreamingBufferFree /home/clear/tmp/suricata/src/util-streaming-buffer.c:148
#4 0x55ace15d52a1 in FileFree /home/clear/tmp/suricata/src/util-file.c:482
#5 0x55ace15d7d8d in FileContainerRecycle /home/clear/tmp/suricata/src/util-file.c:406
#6 0x55ace170711b in suricata::filecontainer::FileContainer::free::ha1ce6bb9d240ac75 src/filecontainer.rs:41
#7 0x55ace170711b in suricata::smb::files::SMBFiles::free::hc5b12d3b0c895ccf src/smb/files.rs:67
#8 0x55ace170711b in suricata::smb::smb::SMBState::free::had064a84c059df8f src/smb/smb.rs:838
#9 0x55ace170711b in rs_smb_state_free src/smb/smb.rs:1791
#10 0x55ace1220603 in AppLayerParserStateCleanup /home/clear/tmp/suricata/src/app-layer-parser.c:1414
#11 0x55ace14056f7 in FlowCleanupAppLayer /home/clear/tmp/suricata/src/flow.c:130
#12 0x55ace14094f6 in FlowClearMemory /home/clear/tmp/suricata/src/flow.c:1041
#13 0x55ace1414e1c in FlowRecycler /home/clear/tmp/suricata/src/flow-manager.c:997
#14 0x55ace15896d5 in TmThreadsManagement /home/clear/tmp/suricata/src/tm-threads.c:706
#15 0x7ff8139fe87e (/usr/lib64/libpthread.so.0+0x987e)
#16 0x7ff81359ae02 in clone (/usr/lib64/haswell/libc.so.6+0x12ce02)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/usr/lib64/libasan.so.5+0x2bdd5)
Thread T12 (FR#01) created by T0 (Suricata-Main) here:
#0 0x7ff813c0caaf in __interceptor_pthread_create (/usr/lib64/libasan.so.5+0x3faaf)
#1 0x55ace158bf06 in TmThreadSpawn /home/clear/tmp/suricata/src/tm-threads.c:1868
#2 0x55ace14187d1 in FlowRecyclerThreadSpawn /home/clear/tmp/suricata/src/flow-manager.c:1076
#3 0x55ace14f897f in RunModeDispatch /home/clear/tmp/suricata/src/runmodes.c:388
#4 0x55ace157a49d in main /home/clear/tmp/suricata/src/suricata.c:3065
#5 0x7ff8134954a1 in __libc_start_main (/usr/lib64/haswell/libc.so.6+0x274a1)
==258009==ABORTING
</pre> Suricata - Bug #3342: Suricata 5.0 crashes while parsing SMB datahttps://redmine.openinfosecfoundation.org/issues/3342?journal_id=147382019-11-23T09:11:17ZVictor Julienvictor@inliniac.net
<ul><li><strong>Target version</strong> changed from <i>5.0.0</i> to <i>5.0.1</i></li><li><strong>Affected Versions</strong> <i>5.0.0</i> added</li></ul> Suricata - Bug #3342: Suricata 5.0 crashes while parsing SMB datahttps://redmine.openinfosecfoundation.org/issues/3342?journal_id=148612019-12-06T14:46:09ZVictor Julienvictor@inliniac.net
<ul></ul><p>Hi Michal, are you capturing the traffic so you can see if you can reproduce it on the pcap?</p> Suricata - Bug #3342: Suricata 5.0 crashes while parsing SMB datahttps://redmine.openinfosecfoundation.org/issues/3342?journal_id=149812019-12-13T10:04:59ZVictor Julienvictor@inliniac.net
<ul><li><strong>Target version</strong> changed from <i>5.0.1</i> to <i>5.0.2</i></li></ul> Suricata - Bug #3342: Suricata 5.0 crashes while parsing SMB datahttps://redmine.openinfosecfoundation.org/issues/3342?journal_id=152212020-02-07T13:47:38ZVictor Julienvictor@inliniac.net
<ul></ul><p>Michal, are you still seeing this with the current master or master-5.0.x branch?</p> Suricata - Bug #3342: Suricata 5.0 crashes while parsing SMB datahttps://redmine.openinfosecfoundation.org/issues/3342?journal_id=152792020-02-13T10:30:51ZVictor Julienvictor@inliniac.net
<ul><li><strong>Target version</strong> changed from <i>5.0.2</i> to <i>5.0.3</i></li></ul> Suricata - Bug #3342: Suricata 5.0 crashes while parsing SMB datahttps://redmine.openinfosecfoundation.org/issues/3342?journal_id=152822020-02-13T10:42:44ZMichal Purzynski
<ul></ul><p>Victor Julien wrote:</p>
<blockquote>
<p>Michal, are you still seeing this with the current master or master-5.0.x branch?</p>
</blockquote>
<p>I can test on -current next week.<br />So far I just filtered out that traffic that was causing crashes and haven't seen anything bad since.</p> Suricata - Bug #3342: Suricata 5.0 crashes while parsing SMB datahttps://redmine.openinfosecfoundation.org/issues/3342?journal_id=155822020-03-16T01:18:02ZMichal Purzynski
<ul></ul><p>Michal Purzynski wrote in <a href="#note-1">#note-1</a>:</p>
<blockquote>
<p>What I see around this time (with Zeek's flow logs and parsed SMB data) is</p>
<p>- there's a couple of connections between 150 - 500 seconds<br />- two connections over 500 seconds have a very high bytes count - over 3.4GB transferred</p>
<p>Connections are multiplexed - there's a long-running TCP connection with multiple SMB operations over it. I can count at least 144 operations.</p>
<p>Out of these 144 SMB operations per one TCP connection</p>
<p>- most are in the range of tens of KBs to 200KBs<br />- there's at least one that literally transfers entire Windows image - 3669480484 bytes of it<br />"Operating Systems\Windows 10 Enterprise x64\Sources\install.wim"</p>
<p>Zeek has no problem tracing that. My guess is that there's some problem with the memory management in conditions like these.</p>
</blockquote>
<p>Michal Purzynski wrote in <a href="#note-11">#note-11</a>:</p>
<blockquote>
<p>Victor Julien wrote:</p>
<blockquote>
<p>Michal, are you still seeing this with the current master or master-5.0.x branch?</p>
</blockquote>
<p>I can test on -current next week.<br />So far I just filtered out that traffic that was causing crashes and haven't seen anything bad since.</p>
</blockquote>
<p>Just deployed Suricata-current (today's git) and am running it with no filters, should have an update within 48h</p>
<p>Suricata 6.0.0-dev (c5cee0516 2020-03-11)</p>
<p>suricata 1491903 1 99 01:13 pts/0 00:03:21 /opt/suricata-current/bin/suricata -vvv -c /etc/suricata/suricata.yaml.local.debug --af-packet<br />suricata 1491904 1 99 01:13 pts/0 00:04:45 /opt/suricata-current/bin/suricata -vvv -c /etc/suricata/suricata.yaml.remote.debug --af-packet</p> Suricata - Bug #3342: Suricata 5.0 crashes while parsing SMB datahttps://redmine.openinfosecfoundation.org/issues/3342?journal_id=157802020-03-30T18:55:35ZVictor Julienvictor@inliniac.net
<ul></ul><p>Hi Michal, are you able provide an update on this?</p> Suricata - Bug #3342: Suricata 5.0 crashes while parsing SMB datahttps://redmine.openinfosecfoundation.org/issues/3342?journal_id=160402020-04-15T00:40:39ZMichal Purzynski
<ul></ul><p>Terrific news - Suricata master has been stable for the past 4+ weeks. It might be I cannot reproduce it anymore.</p> Suricata - Bug #3342: Suricata 5.0 crashes while parsing SMB datahttps://redmine.openinfosecfoundation.org/issues/3342?journal_id=161062020-04-17T07:56:10ZVictor Julienvictor@inliniac.net
<ul><li><strong>Status</strong> changed from <i>Assigned</i> to <i>Closed</i></li><li><strong>Priority</strong> changed from <i>Urgent</i> to <i>Normal</i></li></ul><p>Great, thanks for update Michal!</p>