Feature #1949: only write unique files - Suricata - Open Information Security Foundation

Actions

Feature #1949

closed

Feature #2303: file-store enhancements (aka file-store v2): deduplication; hash-based naming; json metadata and cleanup tooling

only write unique files

Added by Duane Howard about 8 years ago. Updated almost 7 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Jason Ish

Target version:

4.1beta1

Effort:

Difficulty:

Label:

Description

Current behavior for filestore is to extract all. It could be useful to keep state and only write a given file once (maybe per run of Suricata?) For example if 15 users download a popular PE file, we'll end up with 15 copies of the same file on disk. Somewhat related to https://redmine.openinfosecfoundation.org/issues/1948 in that writing to hash for filename would avoid wasted disk space, but not actual time Suricata spends writing files to disk.

Related issues 1 (1 open — 0 closed)

Actions

Copy link

Updated by Duane Howard about 8 years ago

hrm... i meant this to be a feature request, don't appear to be able to change it now?

Actions

Copy link

Updated by Victor Julien about 8 years ago

Tracker changed from Bug to Feature

Actions

Copy link

Updated by Victor Julien about 8 years ago

The file store already starts writing files that are still being transferred. I'm not sure how we can reliably determine duplicate files before we've seen the whole file. In that case we've already started writing it to disk, except perhaps tiny files.

Actions

Copy link