Project

General

Profile

Actions

Feature #1949

closed

Feature #2303: file-store enhancements (aka file-store v2): deduplication; hash-based naming; json metadata and cleanup tooling

only write unique files

Added by Duane Howard almost 5 years ago. Updated over 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Effort:
Difficulty:
Label:

Description

Current behavior for filestore is to extract all. It could be useful to keep state and only write a given file once (maybe per run of Suricata?) For example if 15 users download a popular PE file, we'll end up with 15 copies of the same file on disk. Somewhat related to https://redmine.openinfosecfoundation.org/issues/1948 in that writing to hash for filename would avoid wasted disk space, but not actual time Suricata spends writing files to disk.


Related issues

Related to Task #2309: SuriCon 2017 brainstormNewVictor JulienActions
Actions #1

Updated by Duane Howard almost 5 years ago

hrm... i meant this to be a feature request, don't appear to be able to change it now?

Actions #2

Updated by Victor Julien almost 5 years ago

  • Tracker changed from Bug to Feature
Actions #3

Updated by Victor Julien almost 5 years ago

The file store already starts writing files that are still being transferred. I'm not sure how we can reliably determine duplicate files before we've seen the whole file. In that case we've already started writing it to disk, except perhaps tiny files.

Actions #4

Updated by Andreas Herz over 4 years ago

  • Assignee set to Anonymous
  • Target version set to TBD
Actions #5

Updated by Victor Julien almost 4 years ago

  • Status changed from New to Assigned
  • Assignee changed from Anonymous to Jason Ish
  • Target version changed from TBD to 70

This will be supported if we start using SHA-256 hash for names.

Actions #6

Updated by Jason Ish almost 4 years ago

  • Parent task set to #2303
Actions #7

Updated by Victor Julien almost 4 years ago

  • Related to Task #2309: SuriCon 2017 brainstorm added
Actions #8

Updated by Victor Julien over 3 years ago

  • Status changed from Assigned to Closed
  • Target version changed from 70 to 4.1beta1
Actions

Also available in: Atom PDF