Feature #4579


Add option for 'strict' checking for rules downloaded using the url flag

Added by Ciaran O'Hagan about 1 year ago. Updated 5 months ago.

Target version:


Following up from an email chain with Shivani.

This is a quick description of our use-case: we manage our own rules store, and each day we process new rules from Emerging Threats Open/Pro and deposit them into S3 and distribute keys to our servers running Suricata. During the update deployment, we generate pre-signed URLs from the S3 keys, and use those as parameters to the update application with no-merge enabled. On top of the external rules, we have a collection of custom rules written in-house that we import using the --local flag.

However, in situations where our S3 bucket is empty (or if the S3 keys do not exist), we have found that suricata-update will report an error in the logs, but continues processing and return a 0 exit code:

9/8/2021 -- 13:53:31 - <Info> -- Fetching
9/8/2021 -- 13:53:31 - <Error> -- Failed to fetch HTTP Error 400: Bad Request
9/8/2021 -- 13:53:44 - <Info> -- Fetching
9/8/2021 -- 13:53:44 - <Error> -- Failed to fetch HTTP Error 400: Bad Request
9/8/2021 -- 13:53:44 - <Info> -- Loading local file path/to/custom.rules
9/8/2021 -- 13:53:44 - <Warning> -- No distribution rule directory found.
9/8/2021 -- 13:53:44 - <Info> -- Loaded 74 rules.
9/8/2021 -- 13:53:44 - <Info> -- Disabled 0 rules.
9/8/2021 -- 13:53:44 - <Info> -- Enabled 0 rules.
9/8/2021 -- 13:53:44 - <Info> -- Modified 0 rules.
9/8/2021 -- 13:53:44 - <Info> -- Dropped 0 rules.
9/8/2021 -- 13:53:44 - <Info> -- Enabled 0 rules for flowbit dependencies.
9/8/2021 -- 13:53:44 - <Info> -- Backing up current rules.
9/8/2021 -- 13:53:44 - <Info> -- Writing rule files to directory path/to/suricata/rules: total: 74; enabled: 74; added: 0; removed 0; modified: 0
9/8/2021 -- 13:53:44 - <Info> -- No changes detected, exiting.
$ echo $?

We rely on the return codes during our deployments; and these silent failures have led to misleading conclusions - where our rule deployment "succeeds" but only a very limited number of rules are actually loaded. I've created a workaround on our side to avoid this, but I think it would be great if suricata-update could be extended to include a "strict" flag, which will return an error and "fail loudly" if any of the files provided by the --url flag fail.

Actions #1

Updated by Jason Ish about 1 year ago

  • Target version set to 1.3.0
  • Effort set to low
  • Difficulty set to low

I think we could do 2 things here:

1) If a URL provided by --url fails, then we just fail. As this is provided on the command line, I think we can be more strict about it than sources with an external configuration.


2) Like mentioned, keep the original behaviour and a --fail command line which would be a trigger to convert any errors that might get ignored to fatal, such as a ruleset download link not working.

I'd like to reserve the word strict for possible enhancements around rule parsing as already done by Suricata.

Actions #2

Updated by Shivani Bhardwaj about 1 year ago

  • Status changed from New to Assigned
Actions #3

Updated by Shivani Bhardwaj 9 months ago

  • Tracker changed from Optimization to Feature
  • Status changed from Assigned to In Progress
Actions #4

Updated by Shivani Bhardwaj 9 months ago

  • Status changed from In Progress to In Review
Actions #5

Updated by Jason Ish 5 months ago

  • Status changed from In Review to Closed
  • Target version changed from 1.3.0 to 1.2.4

Also available in: Atom PDF