Suricata

Support domain rollup matching against existing dataset funcitonnality (endswith like behavior)

Option 1 - Domain rollup using specialized Matcher

The matcher would navigate the input buffer string, backward, and for each . (dot) would query the dataset for the presence of the subdomain.
ex:
api.google.com on the inspection buffer:

iterate the string backward, and stop at the first dot:
com -> check the dataset
keep going
google.com -> check the dataset
api.google.com -> check the dataset

It would introduce a new signature keyword:
dns.query; domain-rollup <dataset-name>;

The matcher would automatically automatically perform a dataset:isset internally using the DatasetLookup function directly

Option 2 - Add support to new "domain" Dataset type to enable subdomain matching

Config ex:

datasets:

domain-block:

type: domain
state: domain-block.lst

Signature implementation

dataset:set-> add domain to the associated dataset
dataset:isset-> return true if domain matcher (as described in option 1) find any subdomain in the associated dataset
dataset:isnotset-> return true dataset:isset return false;

In this case the domain could be calculated in reverse order when they are added to the dataset
if we add google.com to the dataset, it would be stored as hash of moc.elgoog
when we navigate the inspection buffer in reverse, it would compute the hash as it move along the char byte array.
upon reaching a . (dot), the hash is ready to be check, no need rehash the string.

Hello François, what is your final goal ? Is the domain keyword like implemented in https://github.com/OISF/suricata/pull/8155 enough for your need ? We still need a discussion on the crate to use but the concept was looking ok for OISF team.

Eric Leblond wrote in #note-3:

Hello François, what is your final goal ? Is the domain keyword like implemented in https://github.com/OISF/suricata/pull/8155 enough for your need ? We still need a discussion on the crate to use but the concept was looking ok for OISF team.

Our end goal is to use datasets to store domain of any subdomain length to match as subdomain.

So we could have IOC like
- test1.com
- test2.test3.com
- test4.test5.test6.com
- any.number.of.subdomains.test7.test8.test9.com

Added to a dataset and allowing to match dns.query/http.host that ends with these subdomain sequence.
We are keen on using dataset because domain can be added/removed very quickly without reloading rules.

The only drawback of this algorithm is that a long domain from the wire like "any.number.of.subdomains.test7.test8.test9.com" will trigger multiple dataset check (8 checks in this case).
But our tests showed that dataset hash check performance is great.

Victor Julien wrote in #note-5:

Hi @Francois Methot is there still interest in this?

We have implemented it and got it working.
It is useful in a very niche use case in IPS mode where new nefarious domains has to be added and activated instantly into suricata without having to wait to compile and reload all the rules.
The instant activation is handled by the usage of datasets. The domain rollup is handled by new custom domain matcher.
We haven't got around to integrate it to the official suricata because it is a tailored solution for the moment.
If there is any interests, I can share the code we have.
You may close that feature/ticket.