Project

General

Profile

Bug #2853

filestore (v1 and v2): dropping of "unwanted" files

Added by magen bluten 3 months ago. Updated 18 days ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
low
Difficulty:
low
Label:

Description

when using the filestore option in combination with e.g. magic filter some files whose magic do not match are dropped as well.

the reason for this is a small bug or typo in FileStoreFileById (util-file.c) and DetectFilestoreMatch (detect-filestore.c).

instead of using the file_track_id the file_store_id is used. file_store_id however is always 0 and only incremented if a file gets dropped. thus
all files of a FileContainer get dropped even if only one file was selected for dropping according to the rules.


Files

suricata.yaml (73.9 KB) suricata.yaml Andreas Herz, 04/17/2019 07:08 AM
extract.pcap (2.75 MB) extract.pcap Andreas Herz, 04/17/2019 07:08 AM
extract-magic.rules (117 Bytes) extract-magic.rules Andreas Herz, 04/17/2019 07:09 AM
extracthttp.pcap (70.4 KB) extracthttp.pcap Andreas Herz, 04/18/2019 08:47 AM

Related issues

Copied to Bug #2966: filestore (v1 and v2): dropping of "unwanted" files (4.1.x)ClosedActions

History

#1

Updated by Andreas Herz about 1 month ago

We could narrow it down to a good reproducible testcase.

Running suricata (even recent git master) on this pcap with filestore v2 enabled will result in the .exe files being stored but also some Windows desktop.ini which won't match the used filemagic string in the rule file.

suricata -c suricata.yaml --runmode autofp -vvv -S extract-magic.rules -r extract.pcap -l /tmp

results in:

file /tmp/files/*/*
files/00/0000000000000000000000000000000000000000000000000000000000000000: PE32 executable (console) Intel 80386, for MS Windows
files/1d/1d4d787047200fc7bcbfc03a496cafda8e49075d2fbf2ff7feab90a4fdea8f89: PE32 executable (console) Intel 80386, for MS Windows
files/1d/1dc15d9d3532d957656f7a16e9c3ad0c91c13b44ac2ab83f4d8fdc02648a2146: PE32 executable (console) Intel 80386 (stripped to external PDB), for MS Windows, UPX compressed
files/23/2365c924112355ddd2d3da985fb09cfc5350f9abc73949c45199c923dab7c40a: Windows desktop.ini
files/4d/4d1c83f5254186d58ce235d0cecd1cc82ff9a3df9f3ed8361c6c173bc426ddd0: Windows desktop.ini
files/88/88aac8a3c7a955e521151ba16b4dc81d9de3e091a76abd19bb4f0e01d572dd5e: Windows desktop.ini
files/a7/a709c2551b8818d7849d31a65446dc2f8c4cca2dcbbc5385604286f49cfdaf1c: Windows desktop.ini
files/be/be41c136b2ac9e3ad69cdd80bbe54a960a436e41f612bbf184a265603b81b745: Windows desktop.ini

With the proposed patch from https://github.com/OISF/suricata/pull/3683 we see the wanted .exe files but the desktop.ini not anymore.

Since the id for a file is always 0 (since file_sort_id is used) a wrong id is used and thus more files from the "container" are stored.

#2

Updated by Peter Manev about 1 month ago

Wondering if it is not somewhat related to - https://redmine.openinfosecfoundation.org/issues/2614 - What do you get if you specify only "filemagic:"executable";" vs "filemagic:"for MS Windows";"?

#3

Updated by Andreas Herz about 1 month ago

This doesn't change the result, only a filemagic string that doesn't match any of the files results in no files stored (obviously :p).
So IMHO not related to your bug.

#4

Updated by Peter Manev about 1 month ago

understood - thanks for checking it out :)

#5

Updated by Andreas Herz about 1 month ago

For reference the same happens with HTTP as well.

alert http any any -> any any (msg:"filestore bug test"; filemagic:"PNG"; filestore; sid:13371337; rev:2;)

results in more files:

files/03/031b2bbeda6fd7e877e50298d2b2ded2073ce6e15f29029b4e50dbd9e81f6be6:                    ASCII text
files/17/17de7185c3cef8064e425b9956c9b2d87cbbd3f6e93917e5c57d1af8d7c25d24:                    UTF-8 Unicode text
files/56/560904cbe632389147334ad588ced6e69f912b3fcc599de56fee7b7d44442c98:                    ASCII text
files/57/57b43ee07432cf8a8b8a17d9d712138194e4564e4b36963a34c495b576b404fe:                    ASCII text
files/66/667cb0b513b1497bee0c2bb633ffd1a6959448d5f9d58d12bb50d9394b3cf543:                    ASCII text
files/76/76ff7909219dfe177a89431965885e7e992e40a2562755ac929f3c8a917a7fe6:                    HTML document, ASCII text
files/7d/7dbe37210602dc0f195c0616e9fc0b2ee652e77f43c95cfb7af9b7d73b900df9:                    ASCII text, with very long lines
files/e0/e092858d5bd66ab33085a966ee4ac0bf0edf6eab8d8b1e66432ee600e904bb4f:                    PNG image data, 36 x 36, 8-bit/color RGBA, non-interlaced
files/e5/e53c64d266a58ab714bcd350d19438017fa0503bd5a3797e7be4bf0d6913e24e:                    ASCII text
files/f7/f7200f61b3285a7deaf0c418c206c94bae135ac3b29977ab7034611407ede45f:                    ASCII text
files/fa/fa65a0bfaa5db268d46b6ba3d8f863dc72c3bf48d8257ec404710e9d0e94aeff:                    ASCII text
#6

Updated by Victor Julien 20 days ago

  • Status changed from New to Assigned
  • Assignee changed from magen bluten to Victor Julien
  • Target version set to 5.0rc1
#7

Updated by Victor Julien 18 days ago

  • Status changed from Assigned to Closed
  • Assignee changed from Victor Julien to magen bluten
  • Priority changed from High to Normal
#8

Updated by Victor Julien 18 days ago

  • Copied to Bug #2966: filestore (v1 and v2): dropping of "unwanted" files (4.1.x) added

Also available in: Atom PDF