Suricata

The more I think about this feature, I think it might be able to assist with the case sensitivity as it relates to some aspects mentioned in https://redmine.openinfosecfoundation.org/issues/5774 and fulfill the need of inspecting individual headers mentioned within https://redmine.openinfosecfoundation.org/issues/5780.

If this optional header-name were made to be case insensitive, it would enable the support of being able to hit on header names of any casing, and allow the individual header inspections.

Just documenting a random thought I had: I've commonly used http.header to chain together multiple headers and their values to create a good fast_pattern . I wonder how fast_patterns would work under this proposed design and if it'd be possible to "chain" together different HTTP header and values into a single fast_pattern.

so if we did

http.header:Access-Control-Allow-Origin; content:"|2a|"; bsize:1; http.header:Access-Control-Allow-Methods; content:"|2a|"; bsize:1; http.header:Access-Control-Allow-Headers; bsize:69; content:"|2a 2c 20|x-"; startswith;

Access-Control-Allow-Origin|3a 20 2a 0d 0a|Access-Control-Allow-Methods|3a 20 2a 0d 0a|Access-Control-Allow-Headers|3a 20 2a 2c 20|x-

I wonder if there would be some way, within the rule language, to indicate the relationship between buffers so that the engine can create that fast_pattern, or perhaps by doing a standalone fast_pattern content and adding nocase to it?

I am not sure I understand what you aim at Brandon...
Is this just syntactic sugar ?
Is it about http header names case normalisation ?

I would say that there are multiple benefits of adopting this model.

• This allows for the use of startswith/endswith/dotprefix/transforms/bsize/etc on specific headers, which currently requires either pcre or header specific keywords (such as http.connection, http.content_type, etc).

• By allowing the engine to determine headers in a case insensitive way, the rule writer doesn't have do silly things in order to match both CamelCased and lower-cased headers when an "exact" match is desired.
  http.header; content:"User-Agent|3a 20|"; nocase; content:"FooBar"; within:6; isdataat:!1,relative
  but instead can do something like
  http.header:user-agent; bsize:6; content:"FooBar";

• instead of having use pcre to limit the inspection to the CommonName field like
  tls.subject; content:"CN="; content:"foobar.com"; distance:0; pcre:"/CN=(?:[^\r\n]+?\.)?foobar\.com(?!\.)/";
  we can use this same idea of an optional field
  tls.subject:cn; dotprefix; content:".foobar.com"; endswith;

• reference: https://redmine.openinfosecfoundation.org/issues/5234

However, the proposed feature is not without issue

• As mentioned above, and reusing the example HTTP response, a really strong fast_pattern would be
  http.header; content:"Access-Control-Allow-Origin|3a 20 2a 0d 0a|Access-Control-Allow-Methods|3a 20 2a 0d 0a|Access-Control-Allow-Headers|3a 20 2a 2c 20|x-"; fast_pattern;
• However this content, unless having nocase added to it, will not match on both HTTP/1 and HTTP/2 traffic. Though adding nocase changes to the detection logic and allows it to match on a X- instead of only the lowercase x- desired. This is to say, the proposed feature does not account for use cases when I want to string together specific headers (case insensitive) and their values (case sensitive) while building a very strong fat_pattern.
• At least one method of addressing this limitation would be to create a relationship between multiple keywords that would allow a fast_pattern to be calculated. I have no idea what this would actually look like, but maybe something like distance/within work today.
  • http.header:access-control-allow-origin; bsize:1; content:"*"; http.header:access-control-allow-methods,within:1; bsize:1; content:"*"; http.header:access-control-allow-headers,within:2; bsize:69; content:"x-"; startswith
  • like I said, i have no idea how this would work or look like, but you can see in my attempt above to put some logic to calculate a relationship between the "dynamic" buffers. http.header_names could also be used to establish this relationship, but i think it might lack some of the granular nature needed to "string" the different contents in order and include the required content match values (* and x- in the above example) when possible.
• another method would be the re-introduction of the :only option to the fast_pattern and add the "nocase" option to it. In my opinion, this is not ideal.
  • http.header; content:"Access-Control-Allow-Origin|3a 20 2a 0d 0a|Access-Control-Allow-Methods|3a 20 2a 0d 0a|Access-Control-Allow-Headers|3a 20 2a 2c 20|x-"; nocase; fast_pattern:only; http.header:access-control-allow-headers; bsize:69; content:"x-"; startswith; pcre:"/^x-[a-f0-9]{64}$/";

Thanks Brandon.

I think https://github.com/OISF/suricata/pull/8621 does allow for the use of startswith/endswith/dotprefix/transforms/bsize/etc on specific headers

It does not address header name case sensitivity
Should it ?

And as you point out, I am not sure about the performance impact...

Philippe Antoine wrote in #note-5:

I think https://github.com/OISF/suricata/pull/8621 does allow for the use of startswith/endswith/dotprefix/transforms/bsize/etc on specific headers

I have to disagree. Because the header name is part of the actual content where as I want the buffer to be filled with just the header value.

What I'm asking for is more comparable to the reason why there is a http.host keyword instead of forcing the use of http.header to inspect there "host" header value. Though, this request allows the header value to be extracted into a buffer to cover any HTTP header name.

Consider the following snippet of an HTTP header
X-Custom-Header: foobar.example.com
under the proposed implimention of request/response header keywords, the following would be required.

As another example... consider how transforms such as to_sah256, stripwhitepsace, etc could be applied to the value of the header only compared to when the header name is included in the buffer being transformed.

These are pretty simple examples, hopefully this helps explain it well enough.

Just another random thought I just had - how would/will negations work when the HTTP header name doesn't exist? reference: https://redmine.openinfosecfoundation.org/issues/2224

Generally speaking, it looks to me that, we have an array of objects (speaking like jsonschema) (example being a HTTP header object which has 2 fields, name and value, both strings)
And we want to match on a specific object for multiple fields...

Or another way to see it, is to have a multi-buffer match restrict the indexes we can match next

Brandon Murphy wrote in #note-8:

Just another random thought I just had - how would/will negations work when the HTTP header name doesn't exist? reference: https://redmine.openinfosecfoundation.org/issues/2224

I was looking into how snort3 addresses HTTP/2 traffic and found they have employed a pretty interesting way of determing if a header name does not exists for both HTTP/1 and HTTP/2.

https://docs.snort.org/rules/options/payload/http/header_test

# check that the User-Agent field is absent
http_header_test:field user-agent,absent;`

When writing #6925 I considered that it might be worth allowing these dynamic sticky buffers to support multi-buffer matching.