Bug #2853: filestore (v1 and v2): dropping of "unwanted" files - Suricata - Open Information Security Foundation

Actions

Copy link

Bug #2853

closed

filestore (v1 and v2): dropping of "unwanted" files

Added by magen bluten almost 6 years ago. Updated over 5 years ago.

Status:

Closed

Priority:

Normal

Assignee:

magen bluten

Target version:

5.0rc1

Affected Versions:

4.1.2

Effort:

low

Difficulty:

low

Label:

Description

when using the filestore option in combination with e.g. magic filter some files whose magic do not match are dropped as well.

the reason for this is a small bug or typo in FileStoreFileById (util-file.c) and DetectFilestoreMatch (detect-filestore.c).

instead of using the file_track_id the file_store_id is used. file_store_id however is always 0 and only incremented if a file gets dropped. thus
all files of a FileContainer get dropped even if only one file was selected for dropping according to the rules.

Files

Download all files

suricata.yaml (73.9 KB) suricata.yaml		Andreas Herz, 04/17/2019 07:08 AM
extract.pcap (2.75 MB) extract.pcap		Andreas Herz, 04/17/2019 07:08 AM
extract-magic.rules (117 Bytes) extract-magic.rules		Andreas Herz, 04/17/2019 07:09 AM
extracthttp.pcap (70.4 KB) extracthttp.pcap		Andreas Herz, 04/18/2019 08:47 AM

Related issues 1 (0 open — 1 closed)

Actions

Copy link Download all files

Updated by Andreas Herz over 5 years ago

File suricata.yaml suricata.yaml added
File extract.pcap extract.pcap added
File extract-magic.rules extract-magic.rules added

We could narrow it down to a good reproducible testcase.

Running suricata (even recent git master) on this pcap with filestore v2 enabled will result in the .exe files being stored but also some Windows desktop.ini which won't match the used filemagic string in the rule file.

suricata -c suricata.yaml --runmode autofp -vvv -S extract-magic.rules -r extract.pcap -l /tmp

results in:

file /tmp/files/*/*
files/00/0000000000000000000000000000000000000000000000000000000000000000: PE32 executable (console) Intel 80386, for MS Windows
files/1d/1d4d787047200fc7bcbfc03a496cafda8e49075d2fbf2ff7feab90a4fdea8f89: PE32 executable (console) Intel 80386, for MS Windows
files/1d/1dc15d9d3532d957656f7a16e9c3ad0c91c13b44ac2ab83f4d8fdc02648a2146: PE32 executable (console) Intel 80386 (stripped to external PDB), for MS Windows, UPX compressed
files/23/2365c924112355ddd2d3da985fb09cfc5350f9abc73949c45199c923dab7c40a: Windows desktop.ini
files/4d/4d1c83f5254186d58ce235d0cecd1cc82ff9a3df9f3ed8361c6c173bc426ddd0: Windows desktop.ini
files/88/88aac8a3c7a955e521151ba16b4dc81d9de3e091a76abd19bb4f0e01d572dd5e: Windows desktop.ini
files/a7/a709c2551b8818d7849d31a65446dc2f8c4cca2dcbbc5385604286f49cfdaf1c: Windows desktop.ini
files/be/be41c136b2ac9e3ad69cdd80bbe54a960a436e41f612bbf184a265603b81b745: Windows desktop.ini

With the proposed patch from https://github.com/OISF/suricata/pull/3683 we see the wanted .exe files but the desktop.ini not anymore.

Since the id for a file is always 0 (since file_sort_id is used) a wrong id is used and thus more files from the "container" are stored.

Actions

Copy link

Updated by Peter Manev over 5 years ago

Wondering if it is not somewhat related to - https://redmine.openinfosecfoundation.org/issues/2614 - What do you get if you specify only "filemagic:"executable";" vs "filemagic:"for MS Windows";"?

Actions

Copy link

Updated by Andreas Herz over 5 years ago

This doesn't change the result, only a filemagic string that doesn't match any of the files results in no files stored (obviously :p).
So IMHO not related to your bug.

Actions

Copy link

Updated by Peter Manev over 5 years ago

understood - thanks for checking it out :)

Actions

Copy link

Updated by Andreas Herz over 5 years ago

File extracthttp.pcap extracthttp.pcap added

For reference the same happens with HTTP as well.

alert http any any -> any any (msg:"filestore bug test"; filemagic:"PNG"; filestore; sid:13371337; rev:2;)

results in more files:

files/03/031b2bbeda6fd7e877e50298d2b2ded2073ce6e15f29029b4e50dbd9e81f6be6:                    ASCII text
files/17/17de7185c3cef8064e425b9956c9b2d87cbbd3f6e93917e5c57d1af8d7c25d24:                    UTF-8 Unicode text
files/56/560904cbe632389147334ad588ced6e69f912b3fcc599de56fee7b7d44442c98:                    ASCII text
files/57/57b43ee07432cf8a8b8a17d9d712138194e4564e4b36963a34c495b576b404fe:                    ASCII text
files/66/667cb0b513b1497bee0c2bb633ffd1a6959448d5f9d58d12bb50d9394b3cf543:                    ASCII text
files/76/76ff7909219dfe177a89431965885e7e992e40a2562755ac929f3c8a917a7fe6:                    HTML document, ASCII text
files/7d/7dbe37210602dc0f195c0616e9fc0b2ee652e77f43c95cfb7af9b7d73b900df9:                    ASCII text, with very long lines
files/e0/e092858d5bd66ab33085a966ee4ac0bf0edf6eab8d8b1e66432ee600e904bb4f:                    PNG image data, 36 x 36, 8-bit/color RGBA, non-interlaced
files/e5/e53c64d266a58ab714bcd350d19438017fa0503bd5a3797e7be4bf0d6913e24e:                    ASCII text
files/f7/f7200f61b3285a7deaf0c418c206c94bae135ac3b29977ab7034611407ede45f:                    ASCII text
files/fa/fa65a0bfaa5db268d46b6ba3d8f863dc72c3bf48d8257ec404710e9d0e94aeff:                    ASCII text

Actions

Copy link