Project

General

Profile

Actions

Bug #1776

closed

Multiple Content-Length headers causes HTP_STREAM_ERROR

Added by David Wharton over 8 years ago. Updated about 5 years ago.

Status:
Closed
Priority:
Normal
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

Summary:

Multiple Content-Length headers causes libhtp to stop parsing and return HTP_STREAM_ERROR.

Example rule:

alert http any any -> any any (msg:"HTTP Host Header"; flow:established, to_server; content:"Host|3A|"; http_header; sid:4567890;)

Example traffic (full pcap attached):

POST /submit.php HTTP/1.1
User-Agent: Mozilla
Host: suricata-ids.org
Content-Length: 45
Cache-Control: no-cache
Content-Length: 45

foo=bar&a=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

Debug output snippet:

htp_connp_REQ_HEADERS: ptr 0x2aec89ee9305 offset 0 len 20
00000000  43 6f 6e 74 65 6e 74 2d  4c 65 6e 67 74 68 3a 20  |Content-Length: |
00000010  34 35 0d 0a                                       |45..|

Header name: ptr 0x1c2c2d58 offset 0 len 14
00000000  43 6f 6e 74 65 6e 74 2d  4c 65 6e 67 74 68        |Content-Length|

Header value: ptr 0x1c2c2d88 offset 0 len 2
00000000  34 35                                             |45|

htp_connp_REQ_HEADERS: ptr 0x2aec89ee9319 offset 0 len 2
00000000  0d 0a                                             |..|

htp_connp_req_data: returning HTP_STREAM_ERROR

Possible solutions:

If all Content-Lengh values are the same, continue as usual. If not, instead of aborting, something else could be done:

  1. Use the first one seen
  2. Use the last one seen
  3. Use the largest
  4. Use the smallest

My first thought is to go with use the smallest.

Notes:

This behavior tested/seen with Suricata 2.0.9 and 3.0.1, not with 1.3.4 though.

To do:

See how other multiple HTTP header names are handled, especially those that populate http_user_agent, http_host, and http_cookie buffers.


Files

multiple_content-length.pcap (558 Bytes) multiple_content-length.pcap David Wharton, 05/02/2016 08:39 AM
duplicate_header-custom.pcap (710 Bytes) duplicate_header-custom.pcap David Wharton, 08/17/2016 10:42 AM

Related issues 1 (0 open1 closed)

Copied to Suricata - Bug #3186: Multiple Content-Length headers causes HTP_STREAM_ERROR (4.1.x)ClosedPhilippe AntoineActions
Actions #1

Updated by Victor Julien over 8 years ago

  • Status changed from New to Assigned
  • Assignee set to Victor Julien
  • Target version set to 70

Sid 2221007 will alert on this.

I'm working on #1656, will see if we can be more liberal about such CL header issues as well.

Actions #2

Updated by David Wharton about 8 years ago

Re: other duplicate headers

When there are duplicate HTTP headers (referring to the header name only, not the value), the normalized buffer (http_header) will only contain the first (top most) one. To match against all of them, the http_raw_header buffer must be used. This may be as designed, I do not know. Attached pcap (duplicate_header-custom.pcap) snippet:

HTTP/1.1 200 OK
Suri-Custom: AAAA
Connection: close
Suri-Custom: BBBB
Content-Length: 45

<html><body><blink>blink</blink></body><html>

The attached pcap can be used with these rules to demonstrate:

alert http any any -> any any (msg:"Suri-Custom HTTP Response Header Detected"; flow: established, to_client; content:"Suri-Custom|3A 20|BBB"; http_header;)

alert http any any -> any any (msg:"Suri-Custom HTTP Response Header Detected"; flow: established, to_client; content:"Suri-Custom|3A 20|BBB"; http_raw_header;)
Actions #3

Updated by Victor Julien about 8 years ago

This seems to be the same issue as #1275. Libhtp merges the 2 records:

 0000  53 75 72 69 2D 43 75 73  74 6F 6D                  Suri-Cus tom
 0000  41 41 41 41 2C 20 42 42  42 42                     AAAA, BB BB

Apparently this mimics the behavior of some HTTP software.

So it sort of functions as intended, at least intended by libhtp originally. I do think it's probably not correct for Suricata, as ppl have an expectation about what http_header, http_user_agent, http_cookie, http_host inspect.

Actions #4

Updated by David Wharton about 8 years ago

Yeah, I forgot to update this ticket after looking in to the issue further the other week. You are correct, when there are multiples of the same header, the values will be concatenated, in the order seen from top to bottom, with a comma and space (", ") between each of them. This is especially worth noting for the normalized individual http buffers - 'http_user_agent', 'http_host', and 'http_cookie'.

Actions #5

Updated by David Wharton about 8 years ago

I forgot to add to my last update that this isn't necessarily a bug, just something to be aware of when writing rules.

Actions #6

Updated by Victor Julien over 5 years ago

  • Assignee changed from Victor Julien to Philippe Antoine
Actions #7

Updated by Philippe Antoine over 5 years ago

Here is a small reminder. The RFC 7230 states :

If a message is received that has multiple Content-Length header
fields with field-values consisting of the same decimal value
, or a
single Content-Length header field with a field value containing a
list of identical decimal values (e.g., "Content-Length: 42, 42"),
indicating that duplicate Content-Length header fields have been
generated or combined by an upstream message processor, then the
recipient MUST either reject the message as invalid or replace the
duplicated field-values with a single valid Content-Length field

containing that decimal value prior to determining the message body
length or forwarding the message.

and

If a message is received without Transfer-Encoding and with
either multiple Content-Length header fields having differing
field-values
or a single Content-Length header field having an
invalid value, then the message framing is invalid and the
recipient MUST treat it as an unrecoverable error.

A pull request is available here :
https://github.com/OISF/libhtp/pull/187

We log only a warning so as to keep parsing.
In case of different values, we take the latest one (same behavior as Wireshark)
Apache returns code 400 (bas request) for a request with two Content-Length headers, even with the same value, and ends the connection.

Actions #8

Updated by Philippe Antoine over 5 years ago

There should be a patch as well in Suricata :

diff --git a/src/app-layer-htp.c b/src/app-layer-htp.c
index d01f420fd..6b331d92f 100644
--- a/src/app-layer-htp.c
+++ b/src/app-layer-htp.c
@@ -511,6 +511,7 @@ struct {
     { "C-E gzip has abnormal value", HTTP_DECODER_EVENT_ABNORMAL_CE_HEADER},
     { "C-E deflate has abnormal value", HTTP_DECODER_EVENT_ABNORMAL_CE_HEADER},
     { "C-E unknown setting", HTTP_DECODER_EVENT_ABNORMAL_CE_HEADER},
+    { "Ambiguous C-L value", HTP_REQUEST_INVALID_C_L},

But I am not sure if this needs a bigger update as a comment states `updated up to libhtp 0.5.7` and we are now at libhtp 0.5.29

Actions #9

Updated by Philippe Antoine over 5 years ago

It should now be ok with :
https://github.com/OISF/libhtp/pull/189
https://github.com/OISF/suricata/pull/3700
https://github.com/OISF/suricata-verify/pull/21

Except that there seems to be a bigger bug : we can bypass the alert by using request pipelining.
My solution would be to make libhtp assign the log->tx variable so that suricata in app-layer-htp.c can assign the app-layer-event to the right transaction id.
But I would prefer someone with a better overview to give a direction.

By the way, why is it the right model to associate an app-layer-event only with the transaction id, and rely on the rules to filter on direction ?

Actions #10

Updated by Philippe Antoine over 5 years ago

Maybe we should wait for https://redmine.openinfosecfoundation.org/issues/1656 as we do not want to break other cases

Actions #11

Updated by Philippe Antoine over 5 years ago

Pull request for libhtp should now be the bigger one
https://github.com/OISF/libhtp/pull/194

Other pull requests (suricata and suricata-verify) should go unchanged

Actions #12

Updated by Victor Julien about 5 years ago

  • Status changed from Assigned to Closed
  • Target version changed from 70 to 5.0rc1
Actions #13

Updated by Victor Julien about 5 years ago

  • Copied to Bug #3186: Multiple Content-Length headers causes HTP_STREAM_ERROR (4.1.x) added
Actions

Also available in: Atom PDF