https://redmine.openinfosecfoundation.org/https://redmine.openinfosecfoundation.org/favicon.ico?17011170022019-03-22T15:42:17ZOpen Information Security FoundationSuricata - Support #2900: alert 'SURICATA STREAM pkt seen on wrong thread' when run mode set to workershttps://redmine.openinfosecfoundation.org/issues/2900?journal_id=116232019-03-22T15:42:17ZVictor Julienvictor@inliniac.net
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-4 priority-4 priority-default" href="/issues/2725">Optimization #2725</a>: stream/packet on wrong thread</i> added</li></ul> Suricata - Support #2900: alert 'SURICATA STREAM pkt seen on wrong thread' when run mode set to workershttps://redmine.openinfosecfoundation.org/issues/2900?journal_id=116242019-03-22T15:55:59ZVictor Julienvictor@inliniac.net
<ul></ul><p>Unfortunately this is a serious issue that can lead to missed alerts and logs. Resolving it should be high priority. If autofp works well I would recommend staying on that. We're tracking the larger issue in <a class="issue tracker-4 status-4 priority-4 priority-default" title="Optimization: stream/packet on wrong thread (Feedback)" href="https://redmine.openinfosecfoundation.org/issues/2725">#2725</a></p> Suricata - Support #2900: alert 'SURICATA STREAM pkt seen on wrong thread' when run mode set to workershttps://redmine.openinfosecfoundation.org/issues/2900?journal_id=119682019-05-03T14:12:40ZAndreas Herzoisf@herzandreas.de
<ul></ul><p>We're trying to narrow this issue down as best as we can. Can you give us more details about your config/setup (I saw a pfsense/netgate post from you, I guess that's related to that?) and the traffic seen?<br />I have similiar issues (but on Linux with AFPacketv3+workers mode) and I'm trying to find a scheme for the traffic that might produce those issues.<br />Thanks</p> Suricata - Support #2900: alert 'SURICATA STREAM pkt seen on wrong thread' when run mode set to workershttps://redmine.openinfosecfoundation.org/issues/2900?journal_id=121282019-05-23T21:33:18ZAndreas Herzoisf@herzandreas.de
<ul><li><strong>Assignee</strong> set to <i>OISF Dev</i></li><li><strong>Target version</strong> set to <i>Support</i></li></ul> Suricata - Support #2900: alert 'SURICATA STREAM pkt seen on wrong thread' when run mode set to workershttps://redmine.openinfosecfoundation.org/issues/2900?journal_id=124742019-06-06T16:06:28ZCooper Nelsoncnelson@ucsd.edu
<ul></ul><p>Adding notes from my recent 'deep dive'.</p>
<p>The root cause appears to be the hardware implementation of RSS in some NICs, confirmed in the ixgbe driver.</p>
<p>Fragmented TCP packets will be hashed by 'sd' only (as the TCP header is only present on the first packet), so fragmented flows will only go to the same queue if <strong>every</strong> TCP packet in the flow is fragmented.</p>
<p>However, in practice its very common for the handshake and first packets of a big TCP flow to not be fragmented and fragmentation occurs later in the flow. Particularly when the packet rates increase due to receive window scaling.</p>
<p>Looking at the documentation for AF_PACKET shows that it is supposed to handle this case properly, but either its not or perhaps suricata isn't setting it properly on all kernels:</p>
<p><a class="external" href="http://man7.org/linux/man-pages/man7/packet.7.html">http://man7.org/linux/man-pages/man7/packet.7.html</a></p>
<p>It also may be the case that this is describing a software implementation that is overridden by hardware RSS, if present. I think I remember regit mentioning that if there was a flow hash generated on the NIC, that is what cluster_flow used.</p>
<p>I do not think it is possible to force a 'sd' hash on the older 10Gbit Intel NICs, however I might be mistaken.</p>
<p>I'm thinking cluster_flow could be modified to handle fragmented TCP packets properly, or simply just hash on 'sd' only. However the TCP packets would still be delivered out-of-order to the worker thread in many cases due to timing issues. Not sure how much of an issue this is with the stream tracker.</p> Suricata - Support #2900: alert 'SURICATA STREAM pkt seen on wrong thread' when run mode set to workershttps://redmine.openinfosecfoundation.org/issues/2900?journal_id=124752019-06-06T16:36:28ZAnonymous
<ul></ul><p>Andreas Herz wrote:</p>
<blockquote>
<p>We're trying to narrow this issue down as best as we can. Can you give us more details about your config/setup (I saw a pfsense/netgate post from you, I guess that's related to that?) and the traffic seen?<br />I have similiar issues (but on Linux with AFPacketv3+workers mode) and I'm trying to find a scheme for the traffic that might produce those issues.<br />Thanks</p>
</blockquote>
<p>I can no longer replicate the issue.<br />I have replicated my (almost) exact same setup from at the time I opened this issue.</p>
<p>Intel pro/1000 PT NIC<br />pfSense 2.4.4-p3 (FreeBSD 11.2)<br />Hardware checksum, tcp and large receive offloading disabled<br />Flow control disabled<br />Suricata 4.1.4_2<br />Netmap + worker mode</p>
<p>Changes:<br />pfSense 2.4.4-p2 -> 2.4.4-p3 (nothing major, still the same FreeBSD release.)<br />Suricata 4.1.2_1 -> 4.1.4_2</p> Suricata - Support #2900: alert 'SURICATA STREAM pkt seen on wrong thread' when run mode set to workershttps://redmine.openinfosecfoundation.org/issues/2900?journal_id=124842019-06-07T16:09:53ZAnonymous
<ul></ul><p>Disregard my last update, issue still persists on FreeBSD 11.2 with netmap and worker mode. Intel pro/1000 PT NIC (em driver).</p> Suricata - Support #2900: alert 'SURICATA STREAM pkt seen on wrong thread' when run mode set to workershttps://redmine.openinfosecfoundation.org/issues/2900?journal_id=128602019-07-09T18:51:22ZAndreas Herzoisf@herzandreas.de
<ul></ul><p>Karel Van Hecke wrote:</p>
<blockquote>
<p>Disregard my last update, issue still persists on FreeBSD 11.2 with netmap and worker mode. Intel pro/1000 PT NIC (em driver).</p>
</blockquote>
<p>Could you check what possible options are offered by the NIC. On Linux we can use ethtool to control relevant parts of that. Not sure how it's done with FreeBSD and especially how this affects netmap. Would be nice to see what options are available.</p> Suricata - Support #2900: alert 'SURICATA STREAM pkt seen on wrong thread' when run mode set to workershttps://redmine.openinfosecfoundation.org/issues/2900?journal_id=174082020-09-05T21:24:04ZAndreas Herzoisf@herzandreas.de
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Closed</i></li></ul><p>Hi, we're closing this issue since there have been no further responses. <br />If you think this bug is still relevant, try to test it again with the <br />most recent version of suricata and reopen the issue. If you want to <br />improve the bug report please take a look at <br /><a class="external" href="https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Reporting_Bugs">https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Reporting_Bugs</a></p> Suricata - Support #2900: alert 'SURICATA STREAM pkt seen on wrong thread' when run mode set to workershttps://redmine.openinfosecfoundation.org/issues/2900?journal_id=263052022-12-30T02:48:13ZMSNYCwaite MSNYCwaite
<ul><li><strong>File</strong> <i>1662.gif</i> added</li></ul> Suricata - Support #2900: alert 'SURICATA STREAM pkt seen on wrong thread' when run mode set to workershttps://redmine.openinfosecfoundation.org/issues/2900?journal_id=263062022-12-30T12:01:31ZVictor Julienvictor@inliniac.net
<ul><li><strong>File</strong> deleted (<del><i>1662.gif</i></del>)</li></ul>