Project

General

Profile

Actions

Feature #5775

open

http.headers - dynamic sticky buffers

Added by Brandon Murphy almost 2 years ago. Updated 7 months ago.

Status:
New
Priority:
High
Assignee:
Target version:
Effort:
Difficulty:
Label:

Description

This idea is largely influenced by Snort 3.0 introduction of of an optional "field name" to the http_header keyword.

Snort's document can be found here: https://docs.snort.org/rules/options/payload/http/header

This request is to create similar support for suricata, which will allow sticky buffers on any http header without the need to add a new specific keyword. This should apply to both request/response headers and support the HTTP/2 overloading of these buffers.
It would also be important that issues where the header is not present though the rule has specific content negated, be handled correctly (please see https://redmine.openinfosecfoundation.org/issues/2224, https://redmine.openinfosecfoundation.org/issues/2479, and https://redmine.openinfosecfoundation.org/issues/4286). Additionally case would need to be considered/normalized. http.header_names can still be used to enforce order as required.

As an example, lets examine the following HTTP traffic from a Lucy Security Phishing HTTP Response

HTTP/1.1 200 OK
Date: Mon, 19 Dec 2022 21:22:19 GMT
Server: Lucy
Set-Cookie: PHPSESSID=t78cghqvdtds1oblk3p38eerj3; path=/; secure; HttpOnly
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: *
Access-Control-Allow-Headers: *, x-c373df213d85a7fb58b442bcfb98407c6f0c58ae50ce6a21aef738c4243f7b4c
Content-Length: 15244
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=UTF-8

I'll key in on the Access-Control-Allow set of headers. Today, if I wanted to write a signature which ensures the value of these headers all start with * and the Access-Control-Allow-Headers matches a specific pattern, while not caring about header order, the following would have to be completed.

http.header; content:"Access-Control-Allow-Origin|3a 20 2a 0d 0a|"; content:"Access-Control-Allow-Methods|3a 20 2a 0d 0a|"; content:"Access-Control-Allow-Headers|3a 20 2a 2c 20|x-"; content:"|0d 0a|"; distance:64; within:2; pcre:"/^Access-Control-Allow-Headers\x3a\x20x-[a-f0-9]{64}\r$/m"

in the proposed, this can be rewritten as

http.header:Access-Control-Allow-Origin; content:"|2a|"; bsize:1; http.header:Access-Control-Allow-Methods; content:"|2a|"; bsize:1; http.header:Access-Control-Allow-Headers; bsize:69; content:"|2a 2c 20|x-"; startswith; pcre:"/^[a-f0-9]{64}/R"

While this is a primitive example, the ability to have sticky buffers on "dynamic" per header basis is a very powerful concept that I'd love to see in Suricata

This logic might also extend to other keywords/protocols, such as TLS and DNS
https://redmine.openinfosecfoundation.org/issues/5234
https://redmine.openinfosecfoundation.org/issues/4227
https://redmine.openinfosecfoundation.org/issues/2448


Related issues 1 (1 open0 closed)

Related to Suricata - Feature #5774: Addressing Mixed Case in HTTP Headers Names and HTTP2NewOISF DevActions
Actions #1

Updated by Brandon Murphy over 1 year ago

The more I think about this feature, I think it might be able to assist with the case sensitivity as it relates to some aspects mentioned in https://redmine.openinfosecfoundation.org/issues/5774 and fulfill the need of inspecting individual headers mentioned within https://redmine.openinfosecfoundation.org/issues/5780.

If this optional header-name were made to be case insensitive, it would enable the support of being able to hit on header names of any casing, and allow the individual header inspections.

Actions #2

Updated by Brandon Murphy over 1 year ago

Just documenting a random thought I had: I've commonly used http.header to chain together multiple headers and their values to create a good fast_pattern . I wonder how fast_patterns would work under this proposed design and if it'd be possible to "chain" together different HTTP header and values into a single fast_pattern.

so if we did

http.header:Access-Control-Allow-Origin; content:"|2a|"; bsize:1; http.header:Access-Control-Allow-Methods; content:"|2a|"; bsize:1; http.header:Access-Control-Allow-Headers; bsize:69; content:"|2a 2c 20|x-"; startswith;

based on the traffic, a strong fast_pattern would be
Access-Control-Allow-Origin|3a 20 2a 0d 0a|Access-Control-Allow-Methods|3a 20 2a 0d 0a|Access-Control-Allow-Headers|3a 20 2a 2c 20|x-

though it would need to include nocase to ensure it met the conditions for HTTP/1 and HTTP/2

I wonder if there would be some way, within the rule language, to indicate the relationship between buffers so that the engine can create that fast_pattern, or perhaps by doing a standalone fast_pattern content and adding nocase to it?

Actions #3

Updated by Philippe Antoine over 1 year ago

I am not sure I understand what you aim at Brandon...
Is this just syntactic sugar ?
Is it about http header names case normalisation ?

Actions #4

Updated by Brandon Murphy over 1 year ago

I would say that there are multiple benefits of adopting this model.

1) Allows for sticky buffers on specific headers without the need to create keywords for each desired header.
  • This allows for the use of startswith/endswith/dotprefix/transforms/bsize/etc on specific headers, which currently requires either pcre or header specific keywords (such as http.connection, http.content_type, etc).
2) Addresses issues with header name case sensitivity for the http.header keyword.
  • By allowing the engine to determine headers in a case insensitive way, the rule writer doesn't have do silly things in order to match both CamelCased and lower-cased headers when an "exact" match is desired.
    http.header; content:"User-Agent|3a 20|"; nocase; content:"FooBar"; within:6; isdataat:!1,relative
    but instead can do something like
    http.header:user-agent; bsize:6; content:"FooBar";
3) This pattern could be used in other protocols such as TLS and DNS (as examples)
  • instead of having use pcre to limit the inspection to the CommonName field like
    tls.subject; content:"CN="; content:"foobar.com"; distance:0; pcre:"/CN=(?:[^\r\n]+?\.)?foobar\.com(?!\.)/";
    we can use this same idea of an optional field
    tls.subject:cn; dotprefix; content:".foobar.com"; endswith;

However, the proposed feature is not without issue

1) fast_pattern
  • As mentioned above, and reusing the example HTTP response, a really strong fast_pattern would be
    http.header; content:"Access-Control-Allow-Origin|3a 20 2a 0d 0a|Access-Control-Allow-Methods|3a 20 2a 0d 0a|Access-Control-Allow-Headers|3a 20 2a 2c 20|x-"; fast_pattern;
  • However this content, unless having nocase added to it, will not match on both HTTP/1 and HTTP/2 traffic. Though adding nocase changes to the detection logic and allows it to match on a X- instead of only the lowercase x- desired. This is to say, the proposed feature does not account for use cases when I want to string together specific headers (case insensitive) and their values (case sensitive) while building a very strong fat_pattern.
  • At least one method of addressing this limitation would be to create a relationship between multiple keywords that would allow a fast_pattern to be calculated. I have no idea what this would actually look like, but maybe something like distance/within work today.
    • http.header:access-control-allow-origin; bsize:1; content:"*"; http.header:access-control-allow-methods,within:1; bsize:1; content:"*"; http.header:access-control-allow-headers,within:2; bsize:69; content:"x-"; startswith
    • like I said, i have no idea how this would work or look like, but you can see in my attempt above to put some logic to calculate a relationship between the "dynamic" buffers. http.header_names could also be used to establish this relationship, but i think it might lack some of the granular nature needed to "string" the different contents in order and include the required content match values (* and x- in the above example) when possible.
  • another method would be the re-introduction of the :only option to the fast_pattern and add the "nocase" option to it. In my opinion, this is not ideal.
    • http.header; content:"Access-Control-Allow-Origin|3a 20 2a 0d 0a|Access-Control-Allow-Methods|3a 20 2a 0d 0a|Access-Control-Allow-Headers|3a 20 2a 2c 20|x-"; nocase; fast_pattern:only; http.header:access-control-allow-headers; bsize:69; content:"x-"; startswith; pcre:"/^x-[a-f0-9]{64}$/";
Actions #5

Updated by Philippe Antoine over 1 year ago

Thanks Brandon.

I think https://github.com/OISF/suricata/pull/8621 does allow for the use of startswith/endswith/dotprefix/transforms/bsize/etc on specific headers

It does not address header name case sensitivity
Should it ?

And as you point out, I am not sure about the performance impact...

Actions #6

Updated by Brandon Murphy over 1 year ago

Philippe Antoine wrote in #note-5:

I think https://github.com/OISF/suricata/pull/8621 does allow for the use of startswith/endswith/dotprefix/transforms/bsize/etc on specific headers

I have to disagree. Because the header name is part of the actual content where as I want the buffer to be filled with just the header value.

What I'm asking for is more comparable to the reason why there is a http.host keyword instead of forcing the use of http.header to inspect there "host" header value. Though, this request allows the header value to be extracted into a buffer to cover any HTTP header name.

Consider the following snippet of an HTTP header
X-Custom-Header: foobar.example.com
under the proposed implimention of request/response header keywords, the following would be required.

http.request_header; content:"X-Custom-Header|3a 20|"; content:"example.com"; endswith; pcre:"(?:[^\r\n]+?\.)?example\.com/$/";
with this feature request completed, something like this would be functionally the same
http.header:x-custom-header; dotprefix; content:".example.com"; endswith;

As another example... consider how transforms such as to_sah256, stripwhitepsace, etc could be applied to the value of the header only compared to when the header name is included in the buffer being transformed.

These are pretty simple examples, hopefully this helps explain it well enough.

Actions #7

Updated by Victor Julien over 1 year ago

  • Assignee changed from OISF Dev to Victor Julien
  • Target version changed from TBD to 8.0.0-beta1

I think it makes sense to support this. I could see other keywords where it might be beneficial as well. Will definitely be a post-7 thing. Will take it into our 8-roadmap discussion.

Actions #8

Updated by Brandon Murphy over 1 year ago

Just another random thought I just had - how would/will negations work when the HTTP header name doesn't exist? reference: https://redmine.openinfosecfoundation.org/issues/2224

Actions #9

Updated by Victor Julien over 1 year ago

  • Related to Feature #5774: Addressing Mixed Case in HTTP Headers Names and HTTP2 added
Actions #10

Updated by Victor Julien over 1 year ago

  • Assignee changed from Victor Julien to OISF Dev
Actions #11

Updated by Philippe Antoine over 1 year ago

Generally speaking, it looks to me that, we have an array of objects (speaking like jsonschema) (example being a HTTP header object which has 2 fields, name and value, both strings)
And we want to match on a specific object for multiple fields...

Or another way to see it, is to have a multi-buffer match restrict the indexes we can match next

Actions #12

Updated by Brandon Murphy over 1 year ago

Brandon Murphy wrote in #note-8:

Just another random thought I just had - how would/will negations work when the HTTP header name doesn't exist? reference: https://redmine.openinfosecfoundation.org/issues/2224

I was looking into how snort3 addresses HTTP/2 traffic and found they have employed a pretty interesting way of determing if a header name does not exists for both HTTP/1 and HTTP/2.

https://docs.snort.org/rules/options/payload/http/header_test

# check that the User-Agent field is absent
http_header_test:field user-agent,absent;`
Actions #13

Updated by Victor Julien about 1 year ago

  • Priority changed from Normal to High
Actions #14

Updated by Brandon Murphy 8 months ago

When writing #6925 I considered that it might be worth allowing these dynamic sticky buffers to support multi-buffer matching.

Actions #15

Updated by Philippe Antoine 7 months ago

We would try to remove SIGMATCH_NOOPT from sigmatch_table[DETECT_HTTP_REQUEST_HEADER].flags |=
SIGMATCH_NOOPT | SIGMATCH_INFO_STICKY_BUFFER;

Then hack DetectHTTPRequestHeaderSetup. to use the optional argument, store it somewhere, and then use it in GetHttp1HeaderData through an added optional ctx argument...

Does not look trivial, but not impossible either...

Actions

Also available in: Atom PDF