Project

General

Profile

Feature #548

Use bloomfilter for filemd5

Added by David André almost 7 years ago. Updated 5 months ago.

Status:
New
Priority:
Low
Target version:
Effort:
low
Difficulty:
medium
Label:

Description

To reduce memory usage, use bloom filters.

Background:
Bloom filters are very memory efficient probabilistic data-structures that dont have false negatives but have false positives.

Pros:
There is already code implemented in suricata source
It is very efficient for blacklists.

Cons:
It might not be efficient for whitelists.

Notes:
Since it has false positives, it would probably be necessary to do a second level validation lookup from data on disk and it will be more expensive.
Implementing through a different keyword (filemd5bloom?) will help avoiding misuse by users.

History

#1

Updated by Victor Julien almost 7 years ago

I think it could make sense for performance, but not for reducing memory. As you mentioned, after a bloom match you will still need to do a validation step. We can't do that from disk, it would be too slow.

#2

Updated by Victor Julien almost 7 years ago

  • Assignee set to Anonymous
  • Target version set to TBD

Performance seems to be OK with the current code as well btw.

#3

Updated by Andreas Herz over 3 years ago

Victor Julien wrote:

Performance seems to be OK with the current code as well btw.

Worth closing the ticket?

#4

Updated by Victor Julien over 3 years ago

No I changed my mind a bit, think it would be interesting to look into this. Still low priority though.

#5

Updated by Victor Julien about 1 year ago

  • Effort set to low
  • Difficulty set to medium
#6

Updated by Andreas Herz 5 months ago

  • Assignee set to Community Ticket

Also available in: Atom PDF