Suricata

Recently discovered that urilen is much faster than bsize applied to http.uri as well.

I'm not 100% sure if this feature, when implemented, would "solve" that.

Using urilen even seems to impact the number of checks a rule experiences. Is it possible that urilen supports some prefilter element that bsize when applied to http.uri does not?

Feel free to let me know if this should be a separate request - it just seems like it might be related.

These do seem related so let's not create a new issue yet.

If bsize setting is the exact length of a pattern, apply startwith/endswith logic. Otherwise, apply it as depth. For dsize (and urilen?) we already do this IIRC.

Is this due to the logic in DetectUrilenApplyToContent?

I've been investigating this and I think that the profiling is attributing too many ticks to the first rule.

I've modified the sids so that the rule using bsize is sid 35 -- from suricata master:

  --------------------------------------------------------------------------
  Date: 4/7/2024 -- 09:57:27. Sorted by: max ticks.
  --------------------------------------------------------------------------
   Num      Rule         Gid      Rev      Ticks        %      Checks   Matches  Max Ticks   Avg Ticks   Avg Match   Avg No Match
  -------- ------------ -------- -------- ------------ ------ -------- -------- ----------- ----------- ----------- --------------
  1        15           1        0        83365        82.15  1        1        83365       83365.00    83365.00    0.00
  2        35           1        0        7501         7.39   1        1        7501        7501.00     7501.00     0.00
  3        20           1        0        6926         6.83   1        1        6926        6926.00     6926.00     0.00
  4        30           1        0        3682         3.63   1        1        3682        3682.00     3682.00     0.00

I added sid 15 -- only dns.query and content:

alert dns any any -> any any (msg:"w/out bsize test"; dns.query; content:"yundol0727.kro.kr";  sid:15;)
alert dns any any -> any any (msg:"depth and isdataat test"; dns.query; content:"yundol0727.kro.kr"; depth:17; isdataat:!1,relative; sid:20;)
alert dns any any -> any any (msg:"depth and endswith test"; dns.query; content:"yundol0727.kro.kr"; depth:17; endswith; sid:30;)
alert dns any any -> any any (msg:"bsize test"; dns.query; content:"yundol0727.kro.kr"; bsize:17; sid:35;)

With modifications to bsize that make it a content modifier under certain circumstances (sid 35):

 --------------------------------------------------------------------------
  Date: 4/7/2024 -- 10:02:41. Sorted by: max ticks.
  --------------------------------------------------------------------------
   Num      Rule         Gid      Rev      Ticks        %      Checks   Matches  Max Ticks   Avg Ticks   Avg Match   Avg No Match
  -------- ------------ -------- -------- ------------ ------ -------- -------- ----------- ----------- ----------- --------------
  1        15           1        0        43219        68.17  1        1        43219       43219.00    43219.00    0.00
  2        20           1        0        7700         12.15  1        1        7700        7700.00     7700.00     0.00
  3        35           1        0        6618         10.44  1        1        6618        6618.00     6618.00     0.00
  4        30           1        0        5860         9.24   1        1        5860        5860.00     5860.00     0.00

Profiling using the same ruleset without sid 15

--------------------------------------------------------------------------
  Date: 4/7/2024 -- 10:14:31. Sorted by: max ticks.
  --------------------------------------------------------------------------
   Num      Rule         Gid      Rev      Ticks        %      Checks   Matches  Max Ticks   Avg Ticks   Avg Match   Avg No Match
  -------- ------------ -------- -------- ------------ ------ -------- -------- ----------- ----------- ----------- --------------
  1        20           1        0        103625       90.32  1        1        103625      103625.00   103625.00   0.00
  2        30           1        0        6522         5.68   1        1        6522        6522.00     6522.00     0.00
  3        35           1        0        4588         4.00   1        1        4588        4588.00     4588.00     0.00

I am continuing to investigate but it appears that the first sid (numerical order) may have extra ticks associated with it.

FWIW, we have observed similar "oddness" within the profiler since the creation of this ticket. We have noticed it enough that at this point we often individually test each rule when comparing variations to determine the fastest method.

I just tested these two rules within our QA environment

alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"UA depth/isdataat test"; flow:established,to_server; http.user_agent; content:"Mozilla"; depth:7; isdataat:!1,relative; sid:1; classtype:unknown; rev:1;)
alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"UA bsize test"; flow:established,to_server; http.user_agent; bsize:7; content:"Mozilla"; sid:1; classtype:unknown; rev:1;)

The total ticks were within ~5% of each other when tested individually, which seems an acceptable margin of error between runs given the scale of total ticks (9.4bn for bsize and and 8.9bn for depth/isdata)

I also went back and tested the urilen vs bsize applied to http.uri and tested them individually.

alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"bsize on http.uri"; flow:established,to_server; http.uri; bsize:17; content:"/js/jquery.min.js"; sid:1; classtype:unknown; rev:1;)
alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"urilen no bsize"; flow:established,to_server; urilen:17; http.uri; content:"/js/jquery.min.js"; sid:1; classtype:unknown; rev:1;)
alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"urilen with bsize"; flow:established,to_server; urilen:17; http.uri; bsize:17; content:"/js/jquery.min.js"; sid:1; classtype:unknown; rev:1;)

urilen had considerably less checks (334 vs 1422), though the avg ticks is actually lower with only the bsize 19040 vs 27k/28k when using urilen. total ticks were in favor of urilen, mostly contributed to the decreased number of checks.

I will note though, that in this specific case where the string being searched for is common within the buffer, but uncommon when is the exact content of the buffer, it is often MUCH faster to use a more generic buffer and matching on a more complete string, than it is to use a more specific buffer.

consider this rule, which had about 4million less checks than the other two above and total ticks were a ~98% reduction.

alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"UA header test"; flow:established,to_server; http.header; content:"User-Agent|3a 20|Mozilla|0d 0a|"; sid:1; classtype:unknown; rev:1;)

----
All of that completed, I do believe the following questions still stand. Perhaps we need another ticket to investigate profiler accuracy?

1. Can you confirm that the optimization that Victor referenced is already being applied when bsize is used?

If bsize setting is the exact length of a pattern, apply startwith/endswith logic. Otherwise, apply it as depth. For dsize (and urilen?) we already do this IIRC.

2. have you been able to find anything that would explain why urilen impacts the number of checks and speed compared to bsize.

I think the first entry being seemly extra expensive is probably just a start up cost. Cold data and/or instruction caches, a memory lookup, that kind of thing. Reordering rules moves the cost to the first rule being evaluated.

"All of that completed, I do believe the following questions still stand. Perhaps we need another ticket to investigate profiler accuracy?

1. Can you confirm that the optimization that Victor referenced is already being applied when bsize is used?

If bsize setting is the exact length of a pattern, apply startwith/endswith logic. Otherwise, apply it as depth. For dsize (and urilen?) we already do this IIRC.

2. have you been able to find anything that would explain why urilen impacts the number of checks and speed compared to bsize."

The bsize optimization is not yet applied in master/main-7.0.x

Do you have a pcap the urilen rule example will trigger on?

Jeff Lucovsky wrote in #note-11:

If bsize setting is the exact length of a pattern, apply startwith/endswith logic. Otherwise, apply it as depth. For dsize (and urilen?) we already do this IIRC.

Sorry, just to be clear, this is currently happening with bsize in master/7.0.x?

No, not yet applied (in development).

Thanks; the urilen is 42; with that change, all 3 rules are evaluated.

Jeff Lucovsky wrote in #note-15:

Thanks; the urilen is 42; with that change, all 3 rules are evaluated.

Correct, to that pcap demonstrates that there is some prefilter logic within urilen that, despite the fast_pattern being the same in all the rules, prevents the rules from being evaluated when the urilen condition is not met.