Bug #1656
closed
several silent bypasses at the HTTP application level (chunking, compression, HTTP 0.9...)
Added by Steffen Ullrich almost 9 years ago.
Updated about 5 years ago.
Description
I'm the author of the HTTP evader test suite which implements various ways to bypass IDS and IPS at the HTTP application level.
While Suricata (contrary to some other major IDS) attempts to properly parse HTTP their are still some bypasses possible.
All tests were done with suricata 2.0.11 on Ubuntu 15.10. The binary was taken from the official PPA on 2016/01/02. The tests were done against a pcap file created by my HTTP evader test tool. All tests inside this pcap file transport the EICAR test virus. Only tests are included where at least one of the major browsers (IE11, Edge, Firefox, Chrome, Safari, Opera) successfully loaded the payload. Since the pcap is too large to attach it will be temporarily available from http://noxxi.de/research/http-evader.pcap.bz2
The following major problems can be seen:
- Transfer-Encoding:
- The code in libhtp explicitly checks for "chunked" and if any other value is given the setting is silently ignored and it continues to check the content-length. Thus it will assume that the content is not chunked if multiple headers are given (maybe junk headers) or if "chunked" is only a substring of the header. The browser behave differently, i.e. Edge and IE use the first header and for the other browsers it is enough if any header is set to "chunked". With Safari it is enough that "chunked" is contained in a substring ("xchunkedy") while Firefox needs at least some word boundary ("x chunked"). Invalid Transfer-Encoding headers are simply ignored by all browsers.
- Chunked Encoding is only defined with HTTP/1.1. Thus a "Transfer-Encoding: chunked" header inside a HTTP/1.0 response must be ignored. Most browsers behave this way but Safari (and squid proxy) ignore the HTTP version restriction.
- Several browsers ignore invalid characters around the size specification of a chunk, i.e. "\vHexsize" (Firefox) or "Hexsize\v" (Firefox, IE, Edge). Suricata fails to parse these sizes and no alerts or other log messages point out the problem. Based on the code in libhtp there should be HTP_LOG_WARNING in this case but I can not find a way to make Suricata log these warnings (eve-log level is the default Info).
- see also http://noxxi.de/research/http-evader-explained-3-chunked.html
- Content-Encoding:
- No support for lzma (Opera). Chrome and Opera also support sdch but I have no tests for this. And with version 44 Firefox will support brotli, but only with https.
- Deflate is only implemented as RFC1951 (raw deflate). But the HTTP specification did actually mean with "deflate" the "zlib" Encoding (RFC1950) and all browsers except IE support this encoding (additionally to raw deflate). See also http://noxxi.de/research/http-evader-explained-2-deflate.html.http://noxxi.de/research/http-evader.html
- Any unsupported encodings are silently ignored (also logged with HTP_LOG_WARNING according to source code of libhtp). This not only means the encodings above but also when the encoding name does not fully match the expected name. Thus browsers accept also "deflate,gzip" which is interpreted as double compression in Firefox and Chrome, single deflate in Safari and no compression in IE and Edge. See also http://noxxi.de/research/http-evader-explained-4-double-encoding.html.
- Decompression errors are ignored too (should also be logged with HTP_LOG_WARNING but nothing shows in the logs). This includes missing or invalid body CRC and length, used of reserved flags and invalid header CRC. But several browsers ignore some or all of these problems, see http://noxxi.de/research/http-evader-explained-5-gzip.html.
- Other problems
- HTTP 0.9 style responses (i.e. no HTTP header) are simply passed through. These responses are accepted by all browsers except Safari. Maybe libhtp would log this too but again I could find no way to make Suricate show the problem. See http://noxxi.de/research/http-evader-explained-1-http09.html
- Most browsers accept a single \r as a delimiter between lines which can be used to hide essential headers, i.e. "HTTP/1.1 200 ok\rTransfer-Encoding: chunked\r\n"
- Suricata assumes that 100, 101, 102 will not have a body. While this should be true browsers behave differently, see http://noxxi.de/research/http-evader-explained-7-lucky-number.html
There are several other problems related to white-space or special characters, junk at the beginning etc. It's best to run all the tests in the pcap file.
For more information about all the tests see http://noxxi.de/research/http-evader.html.
Files
- Status changed from New to Assigned
- Assignee set to Victor Julien
- Target version set to 70
Thanks Steffen, I've been following your research but hadn't found the time to really look into it deeply. The pcap is really helpful.
We'll be splitting this ticket up into the individual issues you have reported. Some may have to go into the libhtp tracker.
The libhtp warnings are generally accessible by enabling the http-events.rules file we ship. Did you have this enabled?
The libhtp warnings are generally accessible by enabling the http-events.rules file we ship. Did you have this enabled?
I've used the default setup in Ubuntu which included the http-event.rules. But, not all of the rules are enabled in the default setup. Once I've enabled these (or alternatively used the http-event.rules coming with the source of suricata 2.0.11) there were the following additional events:
signature_id":2221004,..."SURICATA HTTP invalid response chunk len"
signature_id":2221017,..."SURICATA HTTP invalid response field folding"
The first are the warnings related to invalid chunk sizes I've reported as missing.
The second interestingly comes up if I don't use a final chunk of "0\r\n\r\n" but instead of "0\n \n" (i.e. with space between the newlines).
I still did not get any compression related events even though these are enabled.
I'm looking at the case of HTTP/1.0 response, TE header chunked, non-chunked body (stream 44 in the pcap). If I update libhtp to only consider TE for HTTP/1.1, another stream fails: 41.
This is 41.
GET /chunked/eicar.txt/chunked,http10 HTTP/1.1
Host: evader.example.com
HTTP/1.0 200 ok
Content-type: application/octet-stream
Content-disposition: attachment; filename="eicar.txt"
Transfer-Encoding: chunked
Connection: close
5
X5O!P
5
%@AP[
5
4\PZX
5
54(P^
5
)7CC)
5
7}$EI
5
CAR-S
5
TANDA
5
RD-AN
5
TIVIR
5
US-TE
5
ST-FI
5
LE!$H
3
+H*
0
Are there browsers that dechunk this 41 case?
Victor Julien wrote:
I'm looking at the case of HTTP/1.0 response, TE header chunked, non-chunked body (stream 44 in the pcap). If I update libhtp to only consider TE for HTTP/1.1, another stream fails: 41.
This is 41.
[...]
Are there browsers that dechunk this 41 case?
Unfortunately yes. Safari treats this combination as chunked while the other major browsers don't.
Hmm that is unfortunate indeed. So we need to treat as chunked in that case, but somehow handle the non-chunked case as well. I guess we could try to determine whether the body is chunked based on the first data, but that is going to be tricky especially considering the junk characters some browsers allow before and after the chunk length. Alternatively we could also try parsing it as chunked and fall back to 'plain' body if the parsing fails. Neither is pretty.
Victor Julien wrote:
So we need to treat as chunked in that case, but somehow handle the non-chunked case as well.
In my opinion such data should simply be considered an attack, that is throw an alert or block the data. Since it is known that different endpoints handle this data in a different way chances are very high that this is not just a mistake but an attempt to bypass the IDS. Any attempt to handle such traffic "correctly" would mean that one would need to check several possible interpretations at the same time. This can easily lead to a very high complexity, especially if combined with similar interpretation differences, like:
- If there are multiple Transfer-Encoding headers IE and Edge consider the content chunked only if the last header is "chunked". For all the other browsers it is enough that any header is "chunked".
- Safari and Firefox treat values like "x chunked" the same as "chunked". Other browsers don't.
And of course the behavior of browsers can change. From my experience browsers are mainly designed to treat valid content valid. Then there are a few cases where they treat invalid content explicitly as valid for robustness. And there are lots of cases when they treat invalid content somehow (i.e implementation dependent), simply because they assume that the content is valid but don't check it. See also http://noxxi.de/research/http-evader-explained-10-lazy-browsers.html for some research in this area.
My experiences with flagging protocol anomalies as 'attack' are very poor. Turns out there are many bad implementations out there that violate lots of things. So while I definitely will add ways to alert (and thus block/drop in IPS mode), I'm also going to try to parse as much as we reasonably can.
- Priority changed from Normal to High
- % Done changed from 0 to 30
I've covered about 30 to 40% of the test cases in the pcap now. The first 200 streams now pass, with a few exceptions.
- Assignee changed from Victor Julien to Philippe Antoine
- Priority changed from High to Normal
- Status changed from Assigned to Closed
- Target version changed from 70 to 5.0rc1
Also available in: Atom
PDF