https://redmine.openinfosecfoundation.org/https://redmine.openinfosecfoundation.org/favicon.ico?17011170022016-05-03T09:30:53ZOpen Information Security FoundationSuricata - Bug #1776: Multiple Content-Length headers causes HTP_STREAM_ERRORhttps://redmine.openinfosecfoundation.org/issues/1776?journal_id=67382016-05-03T09:30:53ZVictor Julienvictor@inliniac.net
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Assigned</i></li><li><strong>Assignee</strong> set to <i>Victor Julien</i></li><li><strong>Target version</strong> set to <i>70</i></li></ul><p>Sid 2221007 will alert on this.</p>
<p>I'm working on <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: several silent bypasses at the HTTP application level (chunking, compression, HTTP 0.9...) (Closed)" href="https://redmine.openinfosecfoundation.org/issues/1656">#1656</a>, will see if we can be more liberal about such CL header issues as well.</p> Suricata - Bug #1776: Multiple Content-Length headers causes HTP_STREAM_ERRORhttps://redmine.openinfosecfoundation.org/issues/1776?journal_id=72122016-08-17T10:47:10ZDavid Wharton
<ul><li><strong>File</strong> <a href="/attachments/1254">duplicate_header-custom.pcap</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/1254/duplicate_header-custom.pcap">duplicate_header-custom.pcap</a> added</li></ul><p>Re: other duplicate headers</p>
<p>When there are duplicate HTTP headers (referring to the header name only, not the value), the normalized buffer (http_header) will only contain the first (top most) one. To match against all of them, the http_raw_header buffer must be used. This may be as designed, I do not know. Attached pcap (duplicate_header-custom.pcap) snippet:</p>
<pre>
HTTP/1.1 200 OK
Suri-Custom: AAAA
Connection: close
Suri-Custom: BBBB
Content-Length: 45
<html><body><blink>blink</blink></body><html>
</pre>
<p>The attached pcap can be used with these rules to demonstrate:</p>
<pre>
alert http any any -> any any (msg:"Suri-Custom HTTP Response Header Detected"; flow: established, to_client; content:"Suri-Custom|3A 20|BBB"; http_header;)
alert http any any -> any any (msg:"Suri-Custom HTTP Response Header Detected"; flow: established, to_client; content:"Suri-Custom|3A 20|BBB"; http_raw_header;)
</pre> Suricata - Bug #1776: Multiple Content-Length headers causes HTP_STREAM_ERRORhttps://redmine.openinfosecfoundation.org/issues/1776?journal_id=72402016-08-29T11:13:55ZVictor Julienvictor@inliniac.net
<ul></ul><p>This seems to be the same issue as <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: ET Rule 2003927 not matchin in suricata (Closed)" href="https://redmine.openinfosecfoundation.org/issues/1275">#1275</a>. Libhtp merges the 2 records:<br /><pre>
0000 53 75 72 69 2D 43 75 73 74 6F 6D Suri-Cus tom
0000 41 41 41 41 2C 20 42 42 42 42 AAAA, BB BB
</pre><br />Apparently this mimics the behavior of some HTTP software.</p>
<p>So it sort of functions as intended, at least intended by libhtp originally. I do think it's probably not correct for Suricata, as ppl have an expectation about what http_header, http_user_agent, http_cookie, http_host inspect.</p> Suricata - Bug #1776: Multiple Content-Length headers causes HTP_STREAM_ERRORhttps://redmine.openinfosecfoundation.org/issues/1776?journal_id=72412016-08-29T11:53:33ZDavid Wharton
<ul></ul><p>Yeah, I forgot to update this ticket after looking in to the issue further the other week. You are correct, when there are multiples of the same header, the values will be concatenated, in the order seen from top to bottom, with a comma and space (", ") between each of them. This is especially worth noting for the normalized individual http buffers - 'http_user_agent', 'http_host', and 'http_cookie'.</p> Suricata - Bug #1776: Multiple Content-Length headers causes HTP_STREAM_ERRORhttps://redmine.openinfosecfoundation.org/issues/1776?journal_id=72422016-08-29T11:55:47ZDavid Wharton
<ul></ul><p>I forgot to add to my last update that this isn't necessarily a bug, just something to be aware of when writing rules.</p> Suricata - Bug #1776: Multiple Content-Length headers causes HTP_STREAM_ERRORhttps://redmine.openinfosecfoundation.org/issues/1776?journal_id=113172019-02-28T10:44:24ZVictor Julienvictor@inliniac.net
<ul><li><strong>Assignee</strong> changed from <i>Victor Julien</i> to <i>Philippe Antoine</i></li></ul> Suricata - Bug #1776: Multiple Content-Length headers causes HTP_STREAM_ERRORhttps://redmine.openinfosecfoundation.org/issues/1776?journal_id=113372019-03-04T11:21:35ZPhilippe Antoine
<ul></ul><p>Here is a small reminder. The RFC 7230 states :</p>
<blockquote>
<p>If a message is received that has <ins>multiple Content-Length header<br />fields with field-values consisting of the same decimal value</ins> , or a<br />single Content-Length header field with a field value containing a<br />list of identical decimal values (e.g., "Content-Length: 42, 42"),<br />indicating that duplicate Content-Length header fields have been<br />generated or combined by an upstream message processor, then the<br />recipient MUST <ins>either reject the message as invalid or replace the<br />duplicated field-values with a single valid Content-Length field</ins><br />containing that decimal value prior to determining the message body<br />length or forwarding the message.</p>
</blockquote>
<p>and</p>
<blockquote>
<p>If a message is received without Transfer-Encoding and with<br />either <ins>multiple Content-Length header fields having differing<br />field-values</ins> or a single Content-Length header field having an<br />invalid value, then the message framing is invalid and the<br />recipient <ins>MUST treat it as an unrecoverable error.</ins></p>
</blockquote>
<p>A pull request is available here :<br /><a class="external" href="https://github.com/OISF/libhtp/pull/187">https://github.com/OISF/libhtp/pull/187</a></p>
<p>We log only a warning so as to keep parsing.<br />In case of different values, we take the latest one (same behavior as Wireshark)<br />Apache returns code 400 (bas request) for a request with two Content-Length headers, even with the same value, and ends the connection.</p> Suricata - Bug #1776: Multiple Content-Length headers causes HTP_STREAM_ERRORhttps://redmine.openinfosecfoundation.org/issues/1776?journal_id=113382019-03-04T11:30:02ZPhilippe Antoine
<ul></ul><p>There should be a patch as well in Suricata :</p>
<pre><code class="diff syntaxhl" data-language="diff"><span class="gh">diff --git a/src/app-layer-htp.c b/src/app-layer-htp.c
index d01f420fd..6b331d92f 100644
</span><span class="gd">--- a/src/app-layer-htp.c
</span><span class="gi">+++ b/src/app-layer-htp.c
</span><span class="p">@@ -511,6 +511,7 @@</span> struct {
{ "C-E gzip has abnormal value", HTTP_DECODER_EVENT_ABNORMAL_CE_HEADER},
{ "C-E deflate has abnormal value", HTTP_DECODER_EVENT_ABNORMAL_CE_HEADER},
{ "C-E unknown setting", HTTP_DECODER_EVENT_ABNORMAL_CE_HEADER},
<span class="gi">+ { "Ambiguous C-L value", HTP_REQUEST_INVALID_C_L},
</span></code></pre>
<p>But I am not sure if this needs a bigger update as a comment states `updated up to libhtp 0.5.7` and we are now at libhtp 0.5.29</p> Suricata - Bug #1776: Multiple Content-Length headers causes HTP_STREAM_ERRORhttps://redmine.openinfosecfoundation.org/issues/1776?journal_id=113442019-03-05T10:01:27ZPhilippe Antoine
<ul></ul><p>It should now be ok with :<br /><a class="external" href="https://github.com/OISF/libhtp/pull/189">https://github.com/OISF/libhtp/pull/189</a><br /><a class="external" href="https://github.com/OISF/suricata/pull/3700">https://github.com/OISF/suricata/pull/3700</a><br /><a class="external" href="https://github.com/OISF/suricata-verify/pull/21">https://github.com/OISF/suricata-verify/pull/21</a></p>
<p>Except that there seems to be a bigger bug : we can bypass the alert by using request pipelining.<br />My solution would be to make libhtp assign the log->tx variable so that suricata in app-layer-htp.c can assign the app-layer-event to the right transaction id.<br />But I would prefer someone with a better overview to give a direction.</p>
<p>By the way, why is it the right model to associate an app-layer-event only with the transaction id, and rely on the rules to filter on direction ?</p> Suricata - Bug #1776: Multiple Content-Length headers causes HTP_STREAM_ERRORhttps://redmine.openinfosecfoundation.org/issues/1776?journal_id=113892019-03-08T13:04:02ZPhilippe Antoine
<ul></ul><p>Maybe we should wait for <a class="external" href="https://redmine.openinfosecfoundation.org/issues/1656">https://redmine.openinfosecfoundation.org/issues/1656</a> as we do not want to break other cases</p> Suricata - Bug #1776: Multiple Content-Length headers causes HTP_STREAM_ERRORhttps://redmine.openinfosecfoundation.org/issues/1776?journal_id=117082019-04-02T06:51:00ZPhilippe Antoine
<ul></ul><p>Pull request for libhtp should now be the bigger one <br /><a class="external" href="https://github.com/OISF/libhtp/pull/194">https://github.com/OISF/libhtp/pull/194</a></p>
<p>Other pull requests (suricata and suricata-verify) should go unchanged</p> Suricata - Bug #1776: Multiple Content-Length headers causes HTP_STREAM_ERRORhttps://redmine.openinfosecfoundation.org/issues/1776?journal_id=136252019-09-19T19:29:06ZVictor Julienvictor@inliniac.net
<ul><li><strong>Status</strong> changed from <i>Assigned</i> to <i>Closed</i></li><li><strong>Target version</strong> changed from <i>70</i> to <i>5.0rc1</i></li></ul><p><a class="external" href="https://github.com/OISF/suricata/pull/4218">https://github.com/OISF/suricata/pull/4218</a></p> Suricata - Bug #1776: Multiple Content-Length headers causes HTP_STREAM_ERRORhttps://redmine.openinfosecfoundation.org/issues/1776?journal_id=136812019-09-23T05:24:40ZVictor Julienvictor@inliniac.net
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-1 status-5 priority-4 priority-default closed" href="/issues/3186">Bug #3186</a>: Multiple Content-Length headers causes HTP_STREAM_ERROR (4.1.x)</i> added</li></ul>