Suricata

The libhtp warnings are generally accessible by enabling the http-events.rules file we ship. Did you have this enabled?

I've used the default setup in Ubuntu which included the http-event.rules. But, not all of the rules are enabled in the default setup. Once I've enabled these (or alternatively used the http-event.rules coming with the source of suricata 2.0.11) there were the following additional events:

signature_id":2221004,..."SURICATA HTTP invalid response chunk len" 
signature_id":2221017,..."SURICATA HTTP invalid response field folding" 

The first are the warnings related to invalid chunk sizes I've reported as missing.
The second interestingly comes up if I don't use a final chunk of "0\r\n\r\n" but instead of "0\n \n" (i.e. with space between the newlines).
I still did not get any compression related events even though these are enabled.

I'm looking at the case of HTTP/1.0 response, TE header chunked, non-chunked body (stream 44 in the pcap). If I update libhtp to only consider TE for HTTP/1.1, another stream fails: 41.

This is 41.

GET /chunked/eicar.txt/chunked,http10 HTTP/1.1
Host: evader.example.com

HTTP/1.0 200 ok
Content-type: application/octet-stream
Content-disposition: attachment; filename="eicar.txt" 
Transfer-Encoding: chunked
Connection: close

5
X5O!P
5
%@AP[
5
4\PZX
5
54(P^
5
)7CC)
5
7}$EI
5
CAR-S
5
TANDA
5
RD-AN
5
TIVIR
5
US-TE
5
ST-FI
5
LE!$H
3
+H*
0

Victor Julien wrote:

I'm looking at the case of HTTP/1.0 response, TE header chunked, non-chunked body (stream 44 in the pcap). If I update libhtp to only consider TE for HTTP/1.1, another stream fails: 41.

This is 41.
[...]
Are there browsers that dechunk this 41 case?

Unfortunately yes. Safari treats this combination as chunked while the other major browsers don't.

Hmm that is unfortunate indeed. So we need to treat as chunked in that case, but somehow handle the non-chunked case as well. I guess we could try to determine whether the body is chunked based on the first data, but that is going to be tricky especially considering the junk characters some browsers allow before and after the chunk length. Alternatively we could also try parsing it as chunked and fall back to 'plain' body if the parsing fails. Neither is pretty.

Victor Julien wrote:

So we need to treat as chunked in that case, but somehow handle the non-chunked case as well.

In my opinion such data should simply be considered an attack, that is throw an alert or block the data. Since it is known that different endpoints handle this data in a different way chances are very high that this is not just a mistake but an attempt to bypass the IDS. Any attempt to handle such traffic "correctly" would mean that one would need to check several possible interpretations at the same time. This can easily lead to a very high complexity, especially if combined with similar interpretation differences, like:

• If there are multiple Transfer-Encoding headers IE and Edge consider the content chunked only if the last header is "chunked". For all the other browsers it is enough that any header is "chunked".
• Safari and Firefox treat values like "x chunked" the same as "chunked". Other browsers don't.

And of course the behavior of browsers can change. From my experience browsers are mainly designed to treat valid content valid. Then there are a few cases where they treat invalid content explicitly as valid for robustness. And there are lots of cases when they treat invalid content somehow (i.e implementation dependent), simply because they assume that the content is valid but don't check it. See also http://noxxi.de/research/http-evader-explained-10-lazy-browsers.html for some research in this area.

My experiences with flagging protocol anomalies as 'attack' are very poor. Turns out there are many bad implementations out there that violate lots of things. So while I definitely will add ways to alert (and thus block/drop in IPS mode), I'm also going to try to parse as much as we reasonably can.

These tickets in libhtp:

https://github.com/OISF/libhtp/issues/133
https://github.com/OISF/libhtp/issues/134
https://github.com/OISF/libhtp/issues/135
https://github.com/OISF/libhtp/issues/136
https://github.com/OISF/libhtp/issues/137

Are addressed through https://github.com/OISF/libhtp/pull/139 in libhtp 0.5.20.

https://github.com/inliniac/suricata/pull/2140 adds support for the new libhtp options.

All http evader cases should now work :
https://github.com/OISF/libhtp/pull/194
https://github.com/OISF/suricata-verify/pull/29