Bug #2853


filestore (v1 and v2): dropping of "unwanted" files

Added by magen bluten over 4 years ago. Updated over 4 years ago.

Target version:
Affected Versions:


when using the filestore option in combination with e.g. magic filter some files whose magic do not match are dropped as well.

the reason for this is a small bug or typo in FileStoreFileById (util-file.c) and DetectFilestoreMatch (detect-filestore.c).

instead of using the file_track_id the file_store_id is used. file_store_id however is always 0 and only incremented if a file gets dropped. thus
all files of a FileContainer get dropped even if only one file was selected for dropping according to the rules.


suricata.yaml (73.9 KB) suricata.yaml Andreas Herz, 04/17/2019 07:08 AM
extract.pcap (2.75 MB) extract.pcap Andreas Herz, 04/17/2019 07:08 AM
extract-magic.rules (117 Bytes) extract-magic.rules Andreas Herz, 04/17/2019 07:09 AM
extracthttp.pcap (70.4 KB) extracthttp.pcap Andreas Herz, 04/18/2019 08:47 AM

Related issues 1 (0 open1 closed)

Copied to Suricata - Bug #2966: filestore (v1 and v2): dropping of "unwanted" files (4.1.x)ClosedVictor JulienActions

Updated by Andreas Herz over 4 years ago

We could narrow it down to a good reproducible testcase.

Running suricata (even recent git master) on this pcap with filestore v2 enabled will result in the .exe files being stored but also some Windows desktop.ini which won't match the used filemagic string in the rule file.

suricata -c suricata.yaml --runmode autofp -vvv -S extract-magic.rules -r extract.pcap -l /tmp

results in:

file /tmp/files/*/*
files/00/0000000000000000000000000000000000000000000000000000000000000000: PE32 executable (console) Intel 80386, for MS Windows
files/1d/1d4d787047200fc7bcbfc03a496cafda8e49075d2fbf2ff7feab90a4fdea8f89: PE32 executable (console) Intel 80386, for MS Windows
files/1d/1dc15d9d3532d957656f7a16e9c3ad0c91c13b44ac2ab83f4d8fdc02648a2146: PE32 executable (console) Intel 80386 (stripped to external PDB), for MS Windows, UPX compressed
files/23/2365c924112355ddd2d3da985fb09cfc5350f9abc73949c45199c923dab7c40a: Windows desktop.ini
files/4d/4d1c83f5254186d58ce235d0cecd1cc82ff9a3df9f3ed8361c6c173bc426ddd0: Windows desktop.ini
files/88/88aac8a3c7a955e521151ba16b4dc81d9de3e091a76abd19bb4f0e01d572dd5e: Windows desktop.ini
files/a7/a709c2551b8818d7849d31a65446dc2f8c4cca2dcbbc5385604286f49cfdaf1c: Windows desktop.ini
files/be/be41c136b2ac9e3ad69cdd80bbe54a960a436e41f612bbf184a265603b81b745: Windows desktop.ini

With the proposed patch from we see the wanted .exe files but the desktop.ini not anymore.

Since the id for a file is always 0 (since file_sort_id is used) a wrong id is used and thus more files from the "container" are stored.

Actions #2

Updated by Peter Manev over 4 years ago

Wondering if it is not somewhat related to - - What do you get if you specify only "filemagic:"executable";" vs "filemagic:"for MS Windows";"?

Actions #3

Updated by Andreas Herz over 4 years ago

This doesn't change the result, only a filemagic string that doesn't match any of the files results in no files stored (obviously :p).
So IMHO not related to your bug.

Actions #4

Updated by Peter Manev over 4 years ago

understood - thanks for checking it out :)

Actions #5

Updated by Andreas Herz over 4 years ago

For reference the same happens with HTTP as well.

alert http any any -> any any (msg:"filestore bug test"; filemagic:"PNG"; filestore; sid:13371337; rev:2;)

results in more files:

files/03/031b2bbeda6fd7e877e50298d2b2ded2073ce6e15f29029b4e50dbd9e81f6be6:                    ASCII text
files/17/17de7185c3cef8064e425b9956c9b2d87cbbd3f6e93917e5c57d1af8d7c25d24:                    UTF-8 Unicode text
files/56/560904cbe632389147334ad588ced6e69f912b3fcc599de56fee7b7d44442c98:                    ASCII text
files/57/57b43ee07432cf8a8b8a17d9d712138194e4564e4b36963a34c495b576b404fe:                    ASCII text
files/66/667cb0b513b1497bee0c2bb633ffd1a6959448d5f9d58d12bb50d9394b3cf543:                    ASCII text
files/76/76ff7909219dfe177a89431965885e7e992e40a2562755ac929f3c8a917a7fe6:                    HTML document, ASCII text
files/7d/7dbe37210602dc0f195c0616e9fc0b2ee652e77f43c95cfb7af9b7d73b900df9:                    ASCII text, with very long lines
files/e0/e092858d5bd66ab33085a966ee4ac0bf0edf6eab8d8b1e66432ee600e904bb4f:                    PNG image data, 36 x 36, 8-bit/color RGBA, non-interlaced
files/e5/e53c64d266a58ab714bcd350d19438017fa0503bd5a3797e7be4bf0d6913e24e:                    ASCII text
files/f7/f7200f61b3285a7deaf0c418c206c94bae135ac3b29977ab7034611407ede45f:                    ASCII text
files/fa/fa65a0bfaa5db268d46b6ba3d8f863dc72c3bf48d8257ec404710e9d0e94aeff:                    ASCII text
Actions #6

Updated by Victor Julien over 4 years ago

  • Status changed from New to Assigned
  • Assignee changed from magen bluten to Victor Julien
  • Target version set to 5.0rc1
Actions #7

Updated by Victor Julien over 4 years ago

  • Status changed from Assigned to Closed
  • Assignee changed from Victor Julien to magen bluten
  • Priority changed from High to Normal
Actions #8

Updated by Victor Julien over 4 years ago

  • Copied to Bug #2966: filestore (v1 and v2): dropping of "unwanted" files (4.1.x) added

Also available in: Atom PDF