Suricata

As requested i testet it against the http-events.rules, but they don't trigger either.

Interestingly, it seems that the 2 user-agent headers are merged:

09/16/2014-15:47:52.901650 <hostname unknown> [**] / [**] Autotest httpRequest, HTTPTEST [**] 10.0.13.39:37150 -> 10.0.13.134:80

Looks like libhtp does this unconditionally: https://github.com/ironbee/libhtp/blob/0.5.15/htp/htp_request_generic.c#L66

So is this intended or something that should be "fixed"?

Shall i commit a bug report on github for libhtp or is this already known in upstream libhtp?

Still the same with 3.0 and libhtp 0.5.18 so worth reporting to libhtp upstream?

Still valid issue with 5.0

Well, I see this as about Suricata rule keywords http_header and http_raw_header definition

The rule triggers when we use http_raw_header instead of http_header

The rule does not trigger with http_header keyword because there is a normalization step first
That normalization includes concatenating together the different values for a single header (This is done by libhtp).

That is transforming
User-Agent: Autotest httpRequest
User-Agent: HTTPTEST
into
User-Agent: Autotest httpRequest, HTTPTEST

Maybe the best solution would be to have some keywords name and value that we could use with http_header

I think the normalization is not working in our advantage. I think we should disable the normalization and instead consider this buffer to have 1-N values, instead of just a single one.

This will be quite involved, as it would first require libhtp changes to store this correctly somehow, then in suri we would need to iterate over the 1-N values in detection and logging. The logging (in eve) means we need to change it from scalar to array probably.

then in suri we would need to iterate over the 1-N values in detection

Do you mean that we should then reconstruct N buffers concatenating the one buffered header name with each of the N buffered values ?
Or is there already a motor to do detection across multiple buffers ?

And should we introduce new keywords so as not to break compatibility ?

At suricon we briefly discussed this issue. I tend to see it as a bug rather than a feature. But I think Philippe you mentioned its documented as a feature?

But I think Philippe you mentioned its documented as a feature?

We have the documentation :
https://suricata.readthedocs.io/en/suricata-5.0.0/rules/http-keywords.html#http-header-and-http-raw-header
which says that the header buffer is normalized, but does not explicitly says that this includes concatenating values

But the concatenation is mentioned explicitly for user agent :
https://suricata.readthedocs.io/en/suricata-4.1.4/rules/http-keywords.html#notes

the values will be concatenated in the http_user_agent buffer, in the order seen from top to bottom, with a comma and space (“, “) between each of them.

What do you think about having http_header_name and http_header_value keywords ?
(Maybe complicated for development, but seems useful for signatures writers)