https://redmine.openinfosecfoundation.org/https://redmine.openinfosecfoundation.org/favicon.ico?17011170022019-04-17T07:13:06ZOpen Information Security FoundationSuricata - Bug #2853: filestore (v1 and v2): dropping of "unwanted" fileshttps://redmine.openinfosecfoundation.org/issues/2853?journal_id=118202019-04-17T07:13:06ZAndreas Herzoisf@herzandreas.de
<ul><li><strong>File</strong> <a href="/attachments/1650">suricata.yaml</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/1650/suricata.yaml">suricata.yaml</a> added</li><li><strong>File</strong> <a href="/attachments/1651">extract.pcap</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/1651/extract.pcap">extract.pcap</a> added</li><li><strong>File</strong> <a href="/attachments/1652">extract-magic.rules</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/1652/extract-magic.rules">extract-magic.rules</a> added</li></ul><p>We could narrow it down to a good reproducible testcase.</p>
<p>Running suricata (even recent git master) on this pcap with filestore v2 enabled will result in the .exe files being stored but also some Windows desktop.ini which won't match the used filemagic string in the rule file.</p>
<pre>
suricata -c suricata.yaml --runmode autofp -vvv -S extract-magic.rules -r extract.pcap -l /tmp
</pre>
<p>results in:</p>
<pre>
file /tmp/files/*/*
files/00/0000000000000000000000000000000000000000000000000000000000000000: PE32 executable (console) Intel 80386, for MS Windows
files/1d/1d4d787047200fc7bcbfc03a496cafda8e49075d2fbf2ff7feab90a4fdea8f89: PE32 executable (console) Intel 80386, for MS Windows
files/1d/1dc15d9d3532d957656f7a16e9c3ad0c91c13b44ac2ab83f4d8fdc02648a2146: PE32 executable (console) Intel 80386 (stripped to external PDB), for MS Windows, UPX compressed
files/23/2365c924112355ddd2d3da985fb09cfc5350f9abc73949c45199c923dab7c40a: Windows desktop.ini
files/4d/4d1c83f5254186d58ce235d0cecd1cc82ff9a3df9f3ed8361c6c173bc426ddd0: Windows desktop.ini
files/88/88aac8a3c7a955e521151ba16b4dc81d9de3e091a76abd19bb4f0e01d572dd5e: Windows desktop.ini
files/a7/a709c2551b8818d7849d31a65446dc2f8c4cca2dcbbc5385604286f49cfdaf1c: Windows desktop.ini
files/be/be41c136b2ac9e3ad69cdd80bbe54a960a436e41f612bbf184a265603b81b745: Windows desktop.ini
</pre>
<p>With the proposed patch from <a class="external" href="https://github.com/OISF/suricata/pull/3683">https://github.com/OISF/suricata/pull/3683</a> we see the wanted .exe files but the desktop.ini not anymore.</p>
<p>Since the id for a file is always 0 (since file_sort_id is used) a wrong id is used and thus more files from the "container" are stored.</p> Suricata - Bug #2853: filestore (v1 and v2): dropping of "unwanted" fileshttps://redmine.openinfosecfoundation.org/issues/2853?journal_id=118212019-04-17T08:05:31ZPeter Manevpetermanev@gmail.com
<ul></ul><p>Wondering if it is not somewhat related to - <a class="external" href="https://redmine.openinfosecfoundation.org/issues/2614">https://redmine.openinfosecfoundation.org/issues/2614</a> - What do you get if you specify only "filemagic:"executable";" vs "filemagic:"for MS Windows";"?</p> Suricata - Bug #2853: filestore (v1 and v2): dropping of "unwanted" fileshttps://redmine.openinfosecfoundation.org/issues/2853?journal_id=118222019-04-17T08:17:54ZAndreas Herzoisf@herzandreas.de
<ul></ul><p>This doesn't change the result, only a filemagic string that doesn't match any of the files results in no files stored (obviously :p).<br />So IMHO not related to your bug.</p> Suricata - Bug #2853: filestore (v1 and v2): dropping of "unwanted" fileshttps://redmine.openinfosecfoundation.org/issues/2853?journal_id=118232019-04-17T08:19:42ZPeter Manevpetermanev@gmail.com
<ul></ul><p>understood - thanks for checking it out :)</p> Suricata - Bug #2853: filestore (v1 and v2): dropping of "unwanted" fileshttps://redmine.openinfosecfoundation.org/issues/2853?journal_id=118282019-04-18T08:48:07ZAndreas Herzoisf@herzandreas.de
<ul><li><strong>File</strong> <a href="/attachments/1653">extracthttp.pcap</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/1653/extracthttp.pcap">extracthttp.pcap</a> added</li></ul><p>For reference the same happens with HTTP as well.</p>
<pre>
alert http any any -> any any (msg:"filestore bug test"; filemagic:"PNG"; filestore; sid:13371337; rev:2;)
</pre>
<p>results in more files:</p>
<pre>
files/03/031b2bbeda6fd7e877e50298d2b2ded2073ce6e15f29029b4e50dbd9e81f6be6: ASCII text
files/17/17de7185c3cef8064e425b9956c9b2d87cbbd3f6e93917e5c57d1af8d7c25d24: UTF-8 Unicode text
files/56/560904cbe632389147334ad588ced6e69f912b3fcc599de56fee7b7d44442c98: ASCII text
files/57/57b43ee07432cf8a8b8a17d9d712138194e4564e4b36963a34c495b576b404fe: ASCII text
files/66/667cb0b513b1497bee0c2bb633ffd1a6959448d5f9d58d12bb50d9394b3cf543: ASCII text
files/76/76ff7909219dfe177a89431965885e7e992e40a2562755ac929f3c8a917a7fe6: HTML document, ASCII text
files/7d/7dbe37210602dc0f195c0616e9fc0b2ee652e77f43c95cfb7af9b7d73b900df9: ASCII text, with very long lines
files/e0/e092858d5bd66ab33085a966ee4ac0bf0edf6eab8d8b1e66432ee600e904bb4f: PNG image data, 36 x 36, 8-bit/color RGBA, non-interlaced
files/e5/e53c64d266a58ab714bcd350d19438017fa0503bd5a3797e7be4bf0d6913e24e: ASCII text
files/f7/f7200f61b3285a7deaf0c418c206c94bae135ac3b29977ab7034611407ede45f: ASCII text
files/fa/fa65a0bfaa5db268d46b6ba3d8f863dc72c3bf48d8257ec404710e9d0e94aeff: ASCII text
</pre> Suricata - Bug #2853: filestore (v1 and v2): dropping of "unwanted" fileshttps://redmine.openinfosecfoundation.org/issues/2853?journal_id=119672019-05-03T11:08:26ZVictor Julienvictor@inliniac.net
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Assigned</i></li><li><strong>Assignee</strong> changed from <i>magen bluten</i> to <i>Victor Julien</i></li><li><strong>Target version</strong> set to <i>5.0rc1</i></li></ul> Suricata - Bug #2853: filestore (v1 and v2): dropping of "unwanted" fileshttps://redmine.openinfosecfoundation.org/issues/2853?journal_id=119752019-05-06T05:45:41ZVictor Julienvictor@inliniac.net
<ul><li><strong>Status</strong> changed from <i>Assigned</i> to <i>Closed</i></li><li><strong>Assignee</strong> changed from <i>Victor Julien</i> to <i>magen bluten</i></li><li><strong>Priority</strong> changed from <i>High</i> to <i>Normal</i></li></ul><p><a class="external" href="https://github.com/OISF/suricata/pull/3826">https://github.com/OISF/suricata/pull/3826</a></p> Suricata - Bug #2853: filestore (v1 and v2): dropping of "unwanted" fileshttps://redmine.openinfosecfoundation.org/issues/2853?journal_id=119852019-05-06T08:15:46ZVictor Julienvictor@inliniac.net
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-1 status-5 priority-4 priority-default closed" href="/issues/2966">Bug #2966</a>: filestore (v1 and v2): dropping of "unwanted" files (4.1.x)</i> added</li></ul>