Feature #5775
openhttp.headers - dynamic sticky buffers
Description
This idea is largely influenced by Snort 3.0 introduction of of an optional "field name" to the http_header keyword.
Snort's document can be found here: https://docs.snort.org/rules/options/payload/http/header
This request is to create similar support for suricata, which will allow sticky buffers on any http header without the need to add a new specific keyword. This should apply to both request/response headers and support the HTTP/2 overloading of these buffers.
It would also be important that issues where the header is not present though the rule has specific content negated, be handled correctly (please see https://redmine.openinfosecfoundation.org/issues/2224, https://redmine.openinfosecfoundation.org/issues/2479, and https://redmine.openinfosecfoundation.org/issues/4286). Additionally case would need to be considered/normalized. http.header_names can still be used to enforce order as required.
As an example, lets examine the following HTTP traffic from a Lucy Security Phishing HTTP Response
HTTP/1.1 200 OK Date: Mon, 19 Dec 2022 21:22:19 GMT Server: Lucy Set-Cookie: PHPSESSID=t78cghqvdtds1oblk3p38eerj3; path=/; secure; HttpOnly Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Vary: Accept-Encoding Access-Control-Allow-Origin: * Access-Control-Allow-Methods: * Access-Control-Allow-Headers: *, x-c373df213d85a7fb58b442bcfb98407c6f0c58ae50ce6a21aef738c4243f7b4c Content-Length: 15244 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: text/html; charset=UTF-8
I'll key in on the Access-Control-Allow set of headers. Today, if I wanted to write a signature which ensures the value of these headers all start with * and the Access-Control-Allow-Headers matches a specific pattern, while not caring about header order, the following would have to be completed.
http.header; content:"Access-Control-Allow-Origin|3a 20 2a 0d 0a|"; content:"Access-Control-Allow-Methods|3a 20 2a 0d 0a|"; content:"Access-Control-Allow-Headers|3a 20 2a 2c 20|x-"; content:"|0d 0a|"; distance:64; within:2; pcre:"/^Access-Control-Allow-Headers\x3a\x20x-[a-f0-9]{64}\r$/m"
in the proposed, this can be rewritten as
http.header:Access-Control-Allow-Origin; content:"|2a|"; bsize:1; http.header:Access-Control-Allow-Methods; content:"|2a|"; bsize:1; http.header:Access-Control-Allow-Headers; bsize:69; content:"|2a 2c 20|x-"; startswith; pcre:"/^[a-f0-9]{64}/R"
While this is a primitive example, the ability to have sticky buffers on "dynamic" per header basis is a very powerful concept that I'd love to see in Suricata
This logic might also extend to other keywords/protocols, such as TLS and DNS
https://redmine.openinfosecfoundation.org/issues/5234
https://redmine.openinfosecfoundation.org/issues/4227
https://redmine.openinfosecfoundation.org/issues/2448
Updated by Brandon Murphy over 1 year ago
The more I think about this feature, I think it might be able to assist with the case sensitivity as it relates to some aspects mentioned in https://redmine.openinfosecfoundation.org/issues/5774 and fulfill the need of inspecting individual headers mentioned within https://redmine.openinfosecfoundation.org/issues/5780.
If this optional header-name were made to be case insensitive, it would enable the support of being able to hit on header names of any casing, and allow the individual header inspections.
Updated by Brandon Murphy over 1 year ago
Just documenting a random thought I had: I've commonly used http.header to chain together multiple headers and their values to create a good fast_pattern . I wonder how fast_patterns would work under this proposed design and if it'd be possible to "chain" together different HTTP header and values into a single fast_pattern.
so if we did
http.header:Access-Control-Allow-Origin; content:"|2a|"; bsize:1; http.header:Access-Control-Allow-Methods; content:"|2a|"; bsize:1; http.header:Access-Control-Allow-Headers; bsize:69; content:"|2a 2c 20|x-"; startswith;
based on the traffic, a strong fast_pattern would be
Access-Control-Allow-Origin|3a 20 2a 0d 0a|Access-Control-Allow-Methods|3a 20 2a 0d 0a|Access-Control-Allow-Headers|3a 20 2a 2c 20|x-
though it would need to include
nocase
to ensure it met the conditions for HTTP/1 and HTTP/2
I wonder if there would be some way, within the rule language, to indicate the relationship between buffers so that the engine can create that fast_pattern, or perhaps by doing a standalone fast_pattern content and adding nocase to it?
Updated by Philippe Antoine over 1 year ago
I am not sure I understand what you aim at Brandon...
Is this just syntactic sugar ?
Is it about http header names case normalisation ?
Updated by Brandon Murphy over 1 year ago
I would say that there are multiple benefits of adopting this model.
1) Allows for sticky buffers on specific headers without the need to create keywords for each desired header.- This allows for the use of startswith/endswith/dotprefix/transforms/bsize/etc on specific headers, which currently requires either pcre or header specific keywords (such as http.connection, http.content_type, etc).
- By allowing the engine to determine headers in a case insensitive way, the rule writer doesn't have do silly things in order to match both CamelCased and lower-cased headers when an "exact" match is desired.
http.header; content:"User-Agent|3a 20|"; nocase; content:"FooBar"; within:6; isdataat:!1,relative
but instead can do something likehttp.header:user-agent; bsize:6; content:"FooBar";
- instead of having use pcre to limit the inspection to the CommonName field like
tls.subject; content:"CN="; content:"foobar.com"; distance:0; pcre:"/CN=(?:[^\r\n]+?\.)?foobar\.com(?!\.)/";
we can use this same idea of an optional field
tls.subject:cn; dotprefix; content:".foobar.com"; endswith;
However, the proposed feature is not without issue
1) fast_pattern- As mentioned above, and reusing the example HTTP response, a really strong fast_pattern would be
http.header; content:"Access-Control-Allow-Origin|3a 20 2a 0d 0a|Access-Control-Allow-Methods|3a 20 2a 0d 0a|Access-Control-Allow-Headers|3a 20 2a 2c 20|x-"; fast_pattern;
- However this content, unless having
nocase
added to it, will not match on both HTTP/1 and HTTP/2 traffic. Though addingnocase
changes to the detection logic and allows it to match on aX-
instead of only the lowercasex-
desired. This is to say, the proposed feature does not account for use cases when I want to string together specific headers (case insensitive) and their values (case sensitive) while building a very strong fat_pattern. - At least one method of addressing this limitation would be to create a relationship between multiple keywords that would allow a fast_pattern to be calculated. I have no idea what this would actually look like, but maybe something like distance/within work today.
http.header:access-control-allow-origin; bsize:1; content:"*"; http.header:access-control-allow-methods,within:1; bsize:1; content:"*"; http.header:access-control-allow-headers,within:2; bsize:69; content:"x-"; startswith
- like I said, i have no idea how this would work or look like, but you can see in my attempt above to put some logic to calculate a relationship between the "dynamic" buffers.
http.header_names
could also be used to establish this relationship, but i think it might lack some of the granular nature needed to "string" the different contents in order and include the required content match values (*
andx-
in the above example) when possible.
- another method would be the re-introduction of the
:only
option to the fast_pattern and add the "nocase" option to it. In my opinion, this is not ideal.http.header; content:"Access-Control-Allow-Origin|3a 20 2a 0d 0a|Access-Control-Allow-Methods|3a 20 2a 0d 0a|Access-Control-Allow-Headers|3a 20 2a 2c 20|x-"; nocase; fast_pattern:only; http.header:access-control-allow-headers; bsize:69; content:"x-"; startswith; pcre:"/^x-[a-f0-9]{64}$/";
Updated by Philippe Antoine over 1 year ago
Thanks Brandon.
I think https://github.com/OISF/suricata/pull/8621 does allow for the use of startswith/endswith/dotprefix/transforms/bsize/etc on specific headers
It does not address header name case sensitivity
Should it ?
And as you point out, I am not sure about the performance impact...
Updated by Brandon Murphy over 1 year ago
Philippe Antoine wrote in #note-5:
I think https://github.com/OISF/suricata/pull/8621 does allow for the use of startswith/endswith/dotprefix/transforms/bsize/etc on specific headers
I have to disagree. Because the header name is part of the actual content where as I want the buffer to be filled with just the header value.
What I'm asking for is more comparable to the reason why there is a http.host
keyword instead of forcing the use of http.header
to inspect there "host" header value. Though, this request allows the header value to be extracted into a buffer to cover any HTTP header name.
Consider the following snippet of an HTTP headerX-Custom-Header: foobar.example.com
under the proposed implimention of request/response header keywords, the following would be required.
http.request_header; content:"X-Custom-Header|3a 20|"; content:"example.com"; endswith; pcre:"(?:[^\r\n]+?\.)?example\.com/$/";
with this feature request completed, something like this would be functionally the same
http.header:x-custom-header; dotprefix; content:".example.com"; endswith;
As another example... consider how transforms such as to_sah256, stripwhitepsace, etc could be applied to the value of the header only compared to when the header name is included in the buffer being transformed.
These are pretty simple examples, hopefully this helps explain it well enough.
Updated by Victor Julien over 1 year ago
- Assignee changed from OISF Dev to Victor Julien
- Target version changed from TBD to 8.0.0-beta1
I think it makes sense to support this. I could see other keywords where it might be beneficial as well. Will definitely be a post-7 thing. Will take it into our 8-roadmap discussion.
Updated by Brandon Murphy over 1 year ago
Just another random thought I just had - how would/will negations work when the HTTP header name doesn't exist? reference: https://redmine.openinfosecfoundation.org/issues/2224
Updated by Victor Julien over 1 year ago
- Related to Feature #5774: Addressing Mixed Case in HTTP Headers Names and HTTP2 added
Updated by Victor Julien over 1 year ago
- Assignee changed from Victor Julien to OISF Dev
Updated by Philippe Antoine over 1 year ago
Generally speaking, it looks to me that, we have an array of objects (speaking like jsonschema) (example being a HTTP header object which has 2 fields, name and value, both strings)
And we want to match on a specific object for multiple fields...
Or another way to see it, is to have a multi-buffer match restrict the indexes we can match next
Updated by Brandon Murphy over 1 year ago
Brandon Murphy wrote in #note-8:
Just another random thought I just had - how would/will negations work when the HTTP header name doesn't exist? reference: https://redmine.openinfosecfoundation.org/issues/2224
I was looking into how snort3 addresses HTTP/2 traffic and found they have employed a pretty interesting way of determing if a header name does not exists for both HTTP/1 and HTTP/2.
https://docs.snort.org/rules/options/payload/http/header_test
# check that the User-Agent field is absent http_header_test:field user-agent,absent;`
Updated by Brandon Murphy 8 months ago
When writing #6925 I considered that it might be worth allowing these dynamic sticky buffers to support multi-buffer matching.
Updated by Philippe Antoine 7 months ago
We would try to remove SIGMATCH_NOOPT from sigmatch_table[DETECT_HTTP_REQUEST_HEADER].flags |=
SIGMATCH_NOOPT | SIGMATCH_INFO_STICKY_BUFFER;
Then hack DetectHTTPRequestHeaderSetup
. to use the optional argument, store it somewhere, and then use it in GetHttp1HeaderData
through an added optional ctx argument...
Does not look trivial, but not impossible either...