Summary¶

suricata-update's cache directory (default /var/lib/suricata/update/cache)
never removes an old cached rule-source file once that source's resolved
URL changes. This is the same class of bug as #6763 ("datasets: old
dataset files not being purged"), but in the rule-source fetch path rather
than the dataset path, and doesn't seem to be covered by that fix.

Root cause¶

Cache files are named md5(source_url)-basename(url) (fetch() /
get_tmp_filename() in main.py):

def get_tmp_filename(self, url):
    url_hash = util.md5_hexdigest(url.encode("utf-8"))
    return os.path.join(
        config.get_cache_dir(),
        "%s-%s" % (url_hash, self.url_basename(url)))

Some sources (e.g. et/open, per cache/index.yaml) template the URL with
the installed Suricata version:

url: https://rules.emergingthreats.net/open/suricata-%(__version__)s/emerging.rules.tar.gz

Every time the host's Suricata version changes, %(version)s resolves
to a different string, so the computed hash changes and a brand new cache
file is written. The previous version's cache file is never deleted -
there is no cleanup/prune/gc call anywhere in the fetch path in main.py.

Impact¶

• The cache directory grows without bound across Suricata upgrades, from
  this one source alone.
• suricata-update itself keeps working correctly - it always resolves and
  uses the current file at fetch time, so this isn't a functional break
  for normal operation.
• Anything else that reads the cache directory to reason about "the
  current ruleset" (disk-usage audits, backups, or custom tooling built on
  top of suricata-update's cache) will see this growth, and can
  accidentally surface stale/superseded rule content if it has no way to
  tell which cache file is current for a source.

To reproduce¶

1. Run suricata-update against a state directory that has seen 2+
   different installed Suricata versions over time (e.g. after a couple of
   Suricata upgrades on the same host).
2. Inspect the cache directory:

$ ls /var/lib/suricata/update/cache/*-emerging.rules.tar.gz

On our own server this produced 13 distinct
"<hash>-emerging.rules.tar.gz" files, spanning roughly 2023-09 through the
currently installed version - one per historical Suricata version we'd
run against this state directory. Only 1 of the 13 corresponds to the
currently-installed version's resolved URL; the other 12 are dead weight
that suricata-update will never clean up on its own.

Suggested fix¶

Similar remediation to #6763: either

1. Garbage-collect cache files whose logical source (the un-hashed
   basename) has a newer hash present that matches the currently resolved
   URL for a currently-enabled source, or
2. Key the cache filename off the source's own identifier (e.g. et/open)
   rather than the raw resolved URL, so a Suricata version bump doesn't
   produce a whole new cache identity for the same logical source.

We worked around this client-side in a downstream tool (resolving
cache/index.yaml's url templates + versions.suricata.recommended against
update/sources/*.yaml to identify the current hash per source, then
treating any other same-named cache entry as stale), but this really
belongs in suricata-update itself so every consumer of the cache
directory doesn't have to reimplement that resolution.

Environment¶

• suricata-update version: 1.3.8
• Suricata version(s) involved: Currently 8.0.7, installed from PPA.
• Source demonstrating this: et/open (URL contains %(version)s)

Related: #6763 - same underlying pattern (hashed cache filenames, no
garbage collection), fixed for datasets but apparently not for the
rule-source cache.