Optimization #3427
closedIssue warning/info msg upon datasets of type string that are not base64
Description
Datasets of type string ( https://suricata.readthedocs.io/en/suricata-5.0.1/rules/datasets.html#dataset ) need to be in base64.
Upon loading non base64 strings though there should be a warinng or info message as currently we do not do that but expect the type string to be base64.
cat /opt/rules/dns-requests-dataset.rules pass dns $HOME_NET any -> any any (msg:"StamusN whitelisted domain request"; dns_query; dataset:set,dns.whitelist,type string; bypass; sid:112; rev:1; ) head -5 /etc/suricata/rules/dns.whitelist akadns.net akamai.com akamaihd.net edgekey.net akamai.net suricata -v --af-packet -S "/opt/rules/dns-requests-dataset.rules" -T (suricata.c:1905) <Info> (ParseCommandLine) -- Running suricata under test mode (suricata.c:1084) <Notice> (LogVersion) -- This is Suricata version 5.0.1-dev (4343d1bc0 2019-11-30) running in SYSTEM mode (util-cpu.c:171) <Info> (UtilCpuPrintSummary) -- CPUs/cores online: 112 (util-logopenfile.c:474) <Info> (SCConfLogOpenGeneric) -- eve-log output device (regular) initialized: eve.json (util-logopenfile.c:474) <Info> (SCConfLogOpenGeneric) -- eve-log output device (regular) initialized: /dev/null (output-json-dnp3.c:389) <Info> (OutputDNP3LogInitSub) -- DNP3 log sub-module initialized. (output-tx.c:77) <Notice> (OutputRegisterTxLogger) -- JsonDNP3Log logger not enabled: protocol dnp3 is disabled (output-json-dnp3.c:389) <Info> (OutputDNP3LogInitSub) -- DNP3 log sub-module initialized. (output-tx.c:77) <Notice> (OutputRegisterTxLogger) -- JsonDNP3Log logger not enabled: protocol dnp3 is disabled (util-logopenfile.c:474) <Info> (SCConfLogOpenGeneric) -- stats output device (regular) initialized: stats.log (datasets.c:417) <Notice> (DatasetGet) -- dataset and datarep features are experimental and subject to change (detect-engine-loader.c:353) <Info> (SigLoadSignatures) -- 1 rule files processed. 1 rules successfully loaded, 0 rules failed (util-threshold-config.c:1126) <Info> (SCThresholdConfParseFile) -- Threshold config parsed: 0 rule(s) found (detect-engine-build.c:1416) <Info> (SigAddressPrepareStage1) -- 1 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 1 inspect application layer, 0 are decoder event only (suricata.c:3060) <Notice> (main) -- Configuration provided was successfully loaded. Exiting. (detect-engine-build.c:1716) <Info> (SigAddressCleanupStage1) -- cleaning up signature grouping structure... complete (util-device.c:317) <Notice> (LiveDeviceListClean) -- Stats for 'ens2np0np0': pkts: 0, drop: 0 (-nan%), invalid chksum: 0
Updated by Philippe Antoine 10 months ago
- Assignee set to Shivani Bhardwaj
- Target version set to 8.0.0-beta1
Updated by Shivani Bhardwaj 21 days ago
Hi, Peter!
If we try to load a non-base64 encoded string e.g. curl/7.43.0, it fails during the decoding and gives an error:
Error: datasets: bad base64 encoding
Also note that to us, a string like curl seems like a "non" base64 encoded string but there is no way to know if it is or isn't as it has perfectly valid base64 characters and can be used for decoding purposes which can lead to very unexpected results indeed.
Could you please tell if this makes sense? What is your expectation?
Thank you!
Edit: The strings you have shared in the dataset file do error out now so this was fixed at some point it seems. I'll share which commit. Can't find 4343d1bc0 in the tree..
Updated by Shivani Bhardwaj 19 days ago
- Status changed from In Progress to Closed
This seems to have been fixed in 7.0.4 or earlier. Now, in case of a non base64 encoded string, an error is thrown and the rule is considered invalid.