https://redmine.openinfosecfoundation.org/https://redmine.openinfosecfoundation.org/favicon.ico?17011170022022-08-08T11:50:01ZOpen Information Security FoundationSuricata - Bug #5486: Ethernet metadata is missing for some protocols or parts of a protocolhttps://redmine.openinfosecfoundation.org/issues/5486?journal_id=242482022-08-08T11:50:01ZAndreas Herzoisf@herzandreas.de
<ul></ul><p>more smb example:<br /><pre>
{
"timestamp": "2019-02-15T19:26:50.117953+0100",
"flow_id": 556954253130233,
"pcap_cnt": 52,
"event_type": "smb",
"src_ip": "172.16.10.97",
"src_port": 49892,
"dest_ip": "172.16.10.2",
"dest_port": 445,
"proto": "TCP",
"smb": {
"id": 14,
"dialect": "NT LM 0.12",
"command": "SMB1_COMMAND_NT_TRANS",
"status": "STATUS_SUCCESS",
"status_code": "0x0",
"session_id": 2051,
"tree_id": 4100
},
"ether": {
"src_mac": "00:08:02:1c:47:ae",
"dest_mac": "a4:1f:72:c2:09:6a"
}
}
{
"timestamp": "2019-02-15T19:26:49.915961+0100",
"flow_id": 556954253130233,
"event_type": "smb",
"src_ip": "172.16.10.97",
"src_port": 49892,
"dest_ip": "172.16.10.2",
"dest_port": 445,
"proto": "TCP",
"smb": {
"id": 9,
"dialect": "NT LM 0.12",
"command": "161",
"session_id": 2051,
"tree_id": 4100
},
"ether": {}
}
</pre></p> Suricata - Bug #5486: Ethernet metadata is missing for some protocols or parts of a protocolhttps://redmine.openinfosecfoundation.org/issues/5486?journal_id=242492022-08-08T11:50:23ZVictor Julienvictor@inliniac.net
<ul></ul><p>Can you turn these examples in to SV tests?</p> Suricata - Bug #5486: Ethernet metadata is missing for some protocols or parts of a protocolhttps://redmine.openinfosecfoundation.org/issues/5486?journal_id=242562022-08-09T12:35:53ZAndreas Herzoisf@herzandreas.de
<ul></ul><p>Of course, will do that</p> Suricata - Bug #5486: Ethernet metadata is missing for some protocols or parts of a protocolhttps://redmine.openinfosecfoundation.org/issues/5486?journal_id=242572022-08-09T13:36:53ZAndreas Herzoisf@herzandreas.de
<ul></ul><p>So we narrowed it down to two issues, the first one is rather simple:</p>
<p>The `EveAddCommonOptions` is missing for dcerpc, which is also true for mqtt, quic, pgsql in that version. But this is fixed due to the changes in version 7.0.0. So now all protocols should add the EVE additional information like ethernet and community id (if enabled).</p>
<p>The second one for the smb and http pcap are the same issue with 7.0 but is a bit more complicated. The packets all pass `DecodeEthernet` correctly and the ethernet data is there <strong>but</strong> once they need to be written, some of those run into `CreateJSONEther` with `p->ethh` being `NULL` which shouldn't be since it was present at the decoding state.</p> Suricata - Bug #5486: Ethernet metadata is missing for some protocols or parts of a protocolhttps://redmine.openinfosecfoundation.org/issues/5486?journal_id=271252023-03-23T12:18:43ZRaj S
<ul><li><strong>File</strong> <a href="/attachments/2781">suricata.yaml</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/2781/suricata.yaml">suricata.yaml</a> added</li><li><strong>File</strong> <a href="/attachments/2782">buildinfo.txt</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/2782/buildinfo.txt">buildinfo.txt</a> added</li></ul><p>The reason for the missing ether is due to FlowFinish() creating two<br />pseudo packets p1 and p2 by calling FlowForceReassemblyPseudoPacketGet().</p>
<p>1. FlowForceReassemblyPseudoPacketGet() obtains a new packet p via PacketPoolGetPacket(), and calls<br /> FlowForceReassemblyPseudoPacketSetup()</p>
<p>2. FlowForceReassemblyPseudoPacketSetup() then proceeds to populate 'p' by copying attributes from<br /> the flow 'f'. Crucially, it never copies over the EthernetHdr 'ethh' from the flow to initialise 'p->ethh' because<br /> in the context of the flow struct 'ethh' doesn't exist. The fact that ethh doesn't exist in the flow struct does make sense <br /> to me (ithink?) but basically this is why CreateJSONEther() fails to output an ether value because<br /> 'p->ethh' evaluates to NULL for the newly created pseudo packets. See output-json.c:836.</p>
<p>To further understand the above, run suricata with</p>
<pre><code class="shell syntaxhl" data-language="shell">gdb <span class="nt">-tui</span> <span class="nt">--args</span> suricata/src/.libs/suricata <span class="nt">-r</span> 154.pcap <span class="nt">-c</span> suricata.yaml
</code></pre>
<p>and monitor the following breakpoints</p>
<pre><code class="shell syntaxhl" data-language="shell"><span class="o">(</span>gdb<span class="o">)</span> info breakpoints
Num Type Disp Enb Address What
1 breakpoint keep y 0x00000000003f8f1f <span class="k">in </span>CreateJSONEther at output-json.c:836
2 breakpoint keep y 0x00000000003b3e7c <span class="k">in </span>FlowForceReassemblyPseudoPacketSetup at flow-timeout.c:83
3 breakpoint keep y 0x00000000003b6010 <span class="k">in </span>FlowFinish at flow-worker.c:121
4 breakpoint keep y 0x00000000003b604b <span class="k">in </span>FlowFinish at flow-worker.c:128
5 breakpoint keep y 0x00000000003b60d6 <span class="k">in </span>FlowFinish at flow-worker.c:141
</code></pre>
<p>Given all the above, <br />Using the ugly hack below I can get it to output an ethernet value, but I'm not entirely sure it makes sense or whether it is correct.</p>
<pre><code class="diff syntaxhl" data-language="diff"><span class="gh">diff --git a/src/output-json.c b/src/output-json.c
index f262972..f31e7db 100644
</span><span class="gd">--- a/src/output-json.c
</span><span class="gi">+++ b/src/output-json.c
</span><span class="p">@@ -803,6 +803,7 @@</span> int CreateJSONEther(JsonBuilder *js, const Packet *p, const Flow *f)
/* start new EVE sub-object */
jb_open_object(js, "ether");
if (p == NULL) {
<span class="gi">+invalid_ethh_try_flowpath:;
</span> MacSet *ms = NULL;
/* ensure we have a flow */
if (unlikely(f == NULL)) {
<span class="p">@@ -838,6 +839,8 @@</span> int CreateJSONEther(JsonBuilder *js, const Packet *p, const Flow *f)
uint8_t *dst = p->ethh->eth_dst;
JSONFormatAndAddMACAddr(js, "src_mac", src, false);
JSONFormatAndAddMACAddr(js, "dest_mac", dst, false);
<span class="gi">+ } else {
+ goto invalid_ethh_try_flowpath;
</span> }
}
jb_close(js);
</code></pre> Suricata - Bug #5486: Ethernet metadata is missing for some protocols or parts of a protocolhttps://redmine.openinfosecfoundation.org/issues/5486?journal_id=271322023-03-23T13:24:38ZRaj S
<ul></ul><p>Also note that tunneled traffic exhibits similar behavior.</p>
<p>When TmThreadsSlotVarRun runs the Decode* routines to parse a packet and encounters a<br />tunnel packet, it runs DecodeGRE().</p>
<p>It then calls PacketTunnelPktSetup() to parse the inner packet and calls PacketEnqueueNoLock().<br />In doing so, the newly enqueued packet has no p->ethh.</p>
<p>This causes CreateJSONEther() to fail.</p>
<p>But in this case, I take it that we can't do anything about it given the nature of tunneling?<br />Or does it still make sense to copy over the ethh from the parent packet?</p> Suricata - Bug #5486: Ethernet metadata is missing for some protocols or parts of a protocolhttps://redmine.openinfosecfoundation.org/issues/5486?journal_id=281482023-05-19T14:15:39ZJuliana Fajardini Reichow
<ul><li><strong>Target version</strong> changed from <i>TBD</i> to <i>6.0.13</i></li></ul> Suricata - Bug #5486: Ethernet metadata is missing for some protocols or parts of a protocolhttps://redmine.openinfosecfoundation.org/issues/5486?journal_id=281492023-05-19T14:19:56ZJuliana Fajardini Reichow
<ul><li><strong>Target version</strong> changed from <i>6.0.13</i> to <i>8.0.0-beta1</i></li></ul><p>Hey <a class="user active user-mention" href="https://redmine.openinfosecfoundation.org/users/862">@Sascha Steinbiss</a> would you like to take this one?</p> Suricata - Bug #5486: Ethernet metadata is missing for some protocols or parts of a protocolhttps://redmine.openinfosecfoundation.org/issues/5486?journal_id=281642023-05-19T16:31:47ZSascha Steinbiss
<ul></ul><p>Juliana Fajardini Reichow wrote in <a href="#note-8">#note-8</a>:</p>
<blockquote>
<p>Hey <a class="user active user-mention" href="https://redmine.openinfosecfoundation.org/users/862">@Sascha Steinbiss</a> would you like to take this one?</p>
</blockquote>
<p>Hmm, I am not sure I can currently dedicate the appropriate time to this right now. But it sounds like an attractive learning opportunity to dive into the way packets are passed through Suricata. So... if this needs to be done in time for a release, I'd rather leave it to someone else. If there is time, I might take it.</p> Suricata - Bug #5486: Ethernet metadata is missing for some protocols or parts of a protocolhttps://redmine.openinfosecfoundation.org/issues/5486?journal_id=281832023-05-22T18:14:12ZJuliana Fajardini Reichow
<ul></ul><p>The current timeline for 8.0.0-beta1 is June 4th, would that be doable considering what you've said? If you think it is, we'd be happy to have you work in it! :)</p> Suricata - Bug #5486: Ethernet metadata is missing for some protocols or parts of a protocolhttps://redmine.openinfosecfoundation.org/issues/5486?journal_id=282662023-06-01T07:37:38ZSascha Steinbiss
<ul></ul><p>Unlikely, sorry. I'd rather focus on <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: rfb: parser returns error on unimplemented record types (Closed)" href="https://redmine.openinfosecfoundation.org/issues/5912">#5912</a> as I don't have to dive into unknown code to fix it (and I also kind of told Andreas that I would do it ;))</p> Suricata - Bug #5486: Ethernet metadata is missing for some protocols or parts of a protocolhttps://redmine.openinfosecfoundation.org/issues/5486?journal_id=282702023-06-01T13:07:12ZJuliana Fajardini Reichow
<ul></ul><p>Sascha Steinbiss wrote in <a href="#note-11">#note-11</a>:</p>
<blockquote>
<p>Unlikely, sorry. I'd rather focus on <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: rfb: parser returns error on unimplemented record types (Closed)" href="https://redmine.openinfosecfoundation.org/issues/5912">#5912</a> as I don't have to dive into unknown code to fix it (and I also kind of told Andreas that I would do it ;))</p>
</blockquote>
<p>That's fair! Thanks for working on the other task, and for being responsible about the time you have at the moment. :)</p> Suricata - Bug #5486: Ethernet metadata is missing for some protocols or parts of a protocolhttps://redmine.openinfosecfoundation.org/issues/5486?journal_id=282762023-06-02T11:57:00ZJuliana Fajardini Reichow
<ul></ul><p>Sascha Steinbiss wrote in <a href="#note-11">#note-11</a>:</p>
<blockquote>
<p>Unlikely, sorry. I'd rather focus on <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: rfb: parser returns error on unimplemented record types (Closed)" href="https://redmine.openinfosecfoundation.org/issues/5912">#5912</a> as I don't have to dive into unknown code to fix it (and I also kind of told Andreas that I would do it ;))</p>
</blockquote>
<p>Hey, I messed up when mentioning the release date for 8.0.0-beta1! It's for 2024! Not put pressure in changing your answer, but because my prior message doesn't make a lot of sense, without that part >__<' Sorry about this confusion...</p>