Suricata » Suricata-Update

Following up from an email chain with Shivani.

This is a quick description of our use-case: we manage our own rules store, and each day we process new rules from Emerging Threats Open/Pro and deposit them into S3 and distribute keys to our servers running Suricata. During the update deployment, we generate pre-signed URLs from the S3 keys, and use those as parameters to the update application with no-merge enabled. On top of the external rules, we have a collection of custom rules written in-house that we import using the --local flag.

However, in situations where our S3 bucket is empty (or if the S3 keys do not exist), we have found that suricata-update will report an error in the logs, but continues processing and return a 0 exit code:

9/8/2021 -- 13:53:31 - <Info> -- Fetching https://my.rules.bucket.s3.amazonaws.com/2021-08-05_23/emerging-tftp.rules?X-Amz-Algorithm=...
9/8/2021 -- 13:53:31 - <Error> -- Failed to fetch https://my.rules.bucket.s3.amazonaws.com/2021-08-05_23/emerging-tftp.rules?X-Amz-Algorithm=...: HTTP Error 400: Bad Request
...
9/8/2021 -- 13:53:44 - <Info> -- Fetching https://my.rules.bucket.s3.amazonaws.com/2021-08-05_23/emerging-exploit.rules?X-Amz-Algorithm=...
9/8/2021 -- 13:53:44 - <Error> -- Failed to fetch https://my.rules.bucket.s3.amazonaws.com/2021-08-05_23/emerging-exploit.rulesX-Amz-Algorithm=...: HTTP Error 400: Bad Request
9/8/2021 -- 13:53:44 - <Info> -- Loading local file path/to/custom.rules
9/8/2021 -- 13:53:44 - <Warning> -- No distribution rule directory found.
9/8/2021 -- 13:53:44 - <Info> -- Loaded 74 rules.
9/8/2021 -- 13:53:44 - <Info> -- Disabled 0 rules.
9/8/2021 -- 13:53:44 - <Info> -- Enabled 0 rules.
9/8/2021 -- 13:53:44 - <Info> -- Modified 0 rules.
9/8/2021 -- 13:53:44 - <Info> -- Dropped 0 rules.
9/8/2021 -- 13:53:44 - <Info> -- Enabled 0 rules for flowbit dependencies.
9/8/2021 -- 13:53:44 - <Info> -- Backing up current rules.
9/8/2021 -- 13:53:44 - <Info> -- Writing rule files to directory path/to/suricata/rules: total: 74; enabled: 74; added: 0; removed 0; modified: 0
9/8/2021 -- 13:53:44 - <Info> -- No changes detected, exiting.
$ echo $?
0

We rely on the return codes during our deployments; and these silent failures have led to misleading conclusions - where our rule deployment "succeeds" but only a very limited number of rules are actually loaded. I've created a workaround on our side to avoid this, but I think it would be great if suricata-update could be extended to include a "strict" flag, which will return an error and "fail loudly" if any of the files provided by the --url flag fail.