https://redmine.openinfosecfoundation.org/https://redmine.openinfosecfoundation.org/favicon.ico?17011170022019-03-13T20:02:43ZOpen Information Security FoundationSuricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=115612019-03-13T20:02:43Zchris lujan
<ul></ul><p>Conversely, the http.url field is only matching up until the first space resulting in something like:</p>
<p><code>"http":{"url":"/uid=0(root)"}</code></p>
<p>which leads me to believe those fields are created by splitting the line by spaces.</p> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=116112019-03-22T11:42:54ZVictor Julienvictor@inliniac.net
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Assigned</i></li><li><strong>Assignee</strong> set to <i>Philippe Antoine</i></li><li><strong>Target version</strong> set to <i>TBD</i></li></ul> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=116122019-03-22T11:43:27ZVictor Julienvictor@inliniac.net
<ul></ul><p>I think uri's are not supposed to have spaces, but I think it would be good to address this anyway.</p> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=117122019-04-02T10:36:50ZPhilippe Antoine
<ul></ul><p>Thanks Chris.<br />Indeed, Uris are not supposed to have spaces, but the protocol field is even less supposed to have spaces.<br />So I think we can take the last space in the request line as the uri end, instead of the second one.</p> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=153082020-02-18T08:06:23ZPhilippe Antoine
<ul><li><strong>Related to</strong> <i><a class="issue tracker-5 status-5 priority-4 priority-default closed" href="/issues/3479">Task #3479</a>: libhtp 0.5.33 (4.1.x)</i> added</li></ul> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=153492020-02-18T12:50:54ZPhilippe Antoine
<ul><li><strong>Status</strong> changed from <i>Assigned</i> to <i>In Review</i></li></ul><p><a class="external" href="https://github.com/OISF/libhtp/pull/264">https://github.com/OISF/libhtp/pull/264</a></p> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=160472020-04-15T08:18:12ZPhilippe Antoine
<ul><li><strong>Target version</strong> changed from <i>TBD</i> to <i>6.0.0beta1</i></li></ul> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=169392020-07-13T11:43:02ZPhilippe Antoine
<ul><li><strong>Blocks</strong> <i><a class="issue tracker-5 status-5 priority-4 priority-default closed" href="/issues/3824">Task #3824</a>: libhtp 0.5.34</i> added</li></ul> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=171022020-08-06T07:26:40ZVictor Julienvictor@inliniac.net
<ul><li><strong>Target version</strong> changed from <i>6.0.0beta1</i> to <i>6.0.0rc1</i></li></ul> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=173962020-09-04T14:13:57ZVictor Julienvictor@inliniac.net
<ul><li><strong>Target version</strong> changed from <i>6.0.0rc1</i> to <i>7.0.0-beta1</i></li></ul> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=175182020-09-11T11:04:09ZVictor Julienvictor@inliniac.net
<ul><li><strong>Blocks</strong> deleted (<i><a class="issue tracker-5 status-5 priority-4 priority-default closed" href="/issues/3824">Task #3824</a>: libhtp 0.5.34</i>)</li></ul> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=175272020-09-11T11:30:33ZPhilippe Antoine
<ul><li><strong>Related to</strong> <i><a class="issue tracker-5 status-5 priority-4 priority-default closed" href="/issues/3922">Task #3922</a>: libhtp 0.5.35</i> added</li></ul> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=184402020-11-17T19:40:00ZPhilippe Antoine
<ul><li><strong>Target version</strong> changed from <i>7.0.0-beta1</i> to <i>6.0.1</i></li></ul> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=186032020-11-25T09:46:23ZPhilippe Antoine
<ul><li><strong>Related to</strong> <i><a class="issue tracker-5 status-5 priority-4 priority-default closed" href="/issues/4180">Task #4180</a>: libhtp 0.5.36</i> added</li></ul> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=186082020-11-25T11:10:00ZVictor Julienvictor@inliniac.net
<ul><li><strong>Target version</strong> changed from <i>6.0.1</i> to <i>7.0.0-beta1</i></li></ul> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=186142020-11-25T12:40:02ZPhilippe Antoine
<ul><li><strong>Related to</strong> deleted (<i><a class="issue tracker-5 status-5 priority-4 priority-default closed" href="/issues/4180">Task #4180</a>: libhtp 0.5.36</i>)</li></ul> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=186162020-11-25T14:23:33ZPhilippe Antoine
<ul></ul><p><a class="external" href="https://github.com/OISF/suricata/pull/5599">https://github.com/OISF/suricata/pull/5599</a> for 6.0.1</p>
<p>For 7 :<br />changing the handling in 7 would be good, but I'm not sure it should be optional.</p> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=186962020-12-03T12:46:22ZPhilippe Antoine
<ul></ul><p><a class="external" href="https://github.com/OISF/suricata/pull/5614">https://github.com/OISF/suricata/pull/5614</a> merged for 6.0.1</p>
<p>Still work to do for 7</p> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=205832021-09-08T09:24:03ZPhilippe Antoine
<ul><li><strong>Related to</strong> <i><a class="issue tracker-5 status-5 priority-4 priority-default closed" href="/issues/4667">Task #4667</a>: libhtp 0.5.39</i> added</li></ul> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=232282022-04-25T13:22:47ZPhilippe Antoine
<ul></ul><p><a class="external" href="https://github.com/OISF/suricata/pull/6884">https://github.com/OISF/suricata/pull/6884</a> is latest PR to review</p> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=244592022-09-12T14:11:51ZPhilippe Antoine
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li></ul> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=250682022-10-25T09:16:43ZVictor Julienvictor@inliniac.net
<ul><li><strong>Target version</strong> changed from <i>7.0.0-beta1</i> to <i>7.0.0-rc1</i></li></ul> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=255052022-11-14T08:21:00ZPhilippe Antoine
<ul><li><strong>Status</strong> changed from <i>In Review</i> to <i>Closed</i></li></ul><p><a class="external" href="https://github.com/OISF/suricata/pull/8134">https://github.com/OISF/suricata/pull/8134</a></p> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=265022023-01-24T13:09:33ZVictor Julienvictor@inliniac.net
<ul><li><strong>Status</strong> changed from <i>Closed</i> to <i>In Review</i></li><li><strong>Priority</strong> changed from <i>High</i> to <i>Normal</i></li><li><strong>Target version</strong> changed from <i>7.0.0-rc1</i> to <i>8.0.0-beta1</i></li></ul><p>Was accidentally closed. Postponing once more to give rule writers more time to update things on their end.</p> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=273422023-04-03T07:11:34ZPhilippe Antoine
<ul><li><strong>Target version</strong> changed from <i>8.0.0-beta1</i> to <i>7.0.0-rc2</i></li></ul> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=277922023-05-04T08:48:35ZPhilippe Antoine
<ul></ul><p><a class="external" href="https://github.com/OISF/suricata/pull/8509">https://github.com/OISF/suricata/pull/8509</a> currently</p> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=279762023-05-18T12:07:01ZPhilippe Antoine
<ul><li><strong>Target version</strong> changed from <i>7.0.0-rc2</i> to <i>8.0.0-beta1</i></li></ul> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=285952023-06-15T16:58:41ZBrandon Murphy
<ul><li><strong>File</strong> <a href="/attachments/2816">b8ee56effed96ba.pcap</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/2816/b8ee56effed96ba.pcap">b8ee56effed96ba.pcap</a> added</li></ul><p>I was testing the v16 fork of this and found a difference between 6.0.9 and v16. I was able to confirm the same behavior in v17 fork.</p>
<p>Current Behavior: When the v17 fork is presented with HTTP/1 Request which contains a double space after the URI and before the Protocol, the extra space is added to the end of the URI.</p>
<p>Expected Behavior: I expected the URI would be normalized and remove any trailing spaces, while the http.uri.raw buffer would contain the space.</p>
<pre>
alert http $HOME_NET any -> any any (msg:"Test Double Space after URI - alerts on 6.0.9"; flow:established,to_server; http.method; content:"POST"; http.uri; content:"|2e|php"; endswith; sid:1;)
alert http $HOME_NET any -> any any (msg:"Test Double Space after URI - alerts on v17"; flow:established,to_server; http.method; content:"POST"; http.uri; content:"|2e|php|20|"; endswith; sid:2;)
</pre> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=286302023-06-20T07:25:38ZPhilippe Antoine
<ul></ul><p>Thanks <a class="user active user-mention" href="https://redmine.openinfosecfoundation.org/users/4983">@Brandon Murphy</a> for this report.</p>
<p>Would not the solution rather be to consider the URI before the last block of spaces ? (even the raw one)</p>
<p>Otherwise, <code>SCHTPGenerateNormalizedUri</code> needs to add this normalization step (stripping spaces on the right)</p> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=286562023-06-20T12:31:51ZBrandon Murphy
<ul></ul><blockquote>
<p>Would not the solution rather be to consider the URI before the last block of spaces ? (even the raw one)</p>
</blockquote>
<p>When inconstancies or typos like a double space occur in malicious network traffic, we often use them when creating a rule. The ability to write signatures which can make use of these typos, such as the use of the double space, can provide for good fast_patterns in a rule, along with a low FP rate and the ability to very confidently attribute the traffic to a specific malware family/actor, etc.</p>
<p>I'm not sure where/how the best place to allow that to occur is (if not <code>http.uri.raw</code>). Perhaps <code>http.start</code> is the answer here? Based on <a class="external" href="https://github.com/OISF/suricata/pull/8869">https://github.com/OISF/suricata/pull/8869</a> it appears to have not been selected for "overloading" so that limits our use of <code>http.start</code></p> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=286572023-06-20T12:43:37ZPhilippe Antoine
<ul></ul><blockquote>
<p>When inconstancies or typos like a double space occur in malicious network traffic, we often use them when creating a rule.</p>
</blockquote>
<p>Indeed</p>
<p>Could <code>http.request_line</code> be used in this case ?</p> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=299672023-09-19T08:45:15ZPhilippe Antoine
<ul><li><strong>Subject</strong> changed from <i>http.protocol parsing inaccuracy</i> to <i>http.protocol parsing inaccuracy : accept spaces in URI</i></li></ul> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=305512023-11-07T14:27:49ZBrandon Murphy
<ul></ul><p>Sorry, it looks like I dropped the ball on this one. I've "watched" this ticket to make sure that doesn't happen again.</p>
<blockquote>
<p>Could http.request_line be used in this case ?</p>
</blockquote>
<p>I guess if this is the only solution, then ok. The issue is that it's uncommon usage of the buffer and lacks normalization of the uri, etc. It would pretty much make http.request_line the new http.uri.raw in cases where a double space is present, which will be an edge case often forgotten about and hard to teach people new to the engine. At the very least, if this is selected as the desired option, would it be possible to add a section to the documents on this behavior?</p>
<p>I feel like anyone looking at network traffic can clearly see that the URI has a space in it and that the protocol is at the end.</p>
<blockquote>
<p>Would not the solution rather be to consider the URI before the last block of spaces ? (even the raw one)</p>
</blockquote>
<p>RFC says this<br /><pre>
A request-line begins with a method token, followed by a single space
(SP), the request-target, another single space (SP), the protocol
version, and ends with CRLF.
request-line = method SP request-target SP HTTP-version CRLF
</pre></p>
<p>What do you think about considering the URI before the last space? (not the last "block of spaces"). This conforms with format of the request-line as per the RFC.</p>
<p>given a request line of <code>GET /foo.php HTTP/1.1</code></p>
<p>http.uri = <code>/foo.php</code> (no trailing space, gets normalized out)<br />http.uri.raw = <code>/foo.php </code> (trailing space)<br />http.request_line = <code>GET /foo.php HTTP/1.1</code> (contains the double space)</p>
<p>More or less the method would be anything to the "left" of the first space, the version anything "right" of the last space and anything in between those two is the URI?</p> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=305522023-11-07T14:28:08ZBrandon Murphy
<ul></ul><p><a class="external" href="https://forum.suricata.io/t/http-protocol-error-field-parsing/4150/2">https://forum.suricata.io/t/http-protocol-error-field-parsing/4150/2</a></p>
<p>Issue mentioned here</p> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=305532023-11-07T14:45:59ZPhilippe Antoine
<ul></ul><p>So, <code>SCHTPGenerateNormalizedUri</code> needs to add this normalization step (stripping spaces on the right), is it correct <a class="user active user-mention" href="https://redmine.openinfosecfoundation.org/users/4983">@Brandon Murphy</a> ?</p> Suricata - Bug #2881: http.protocol parsing inaccuracy : accept spaces in URIhttps://redmine.openinfosecfoundation.org/issues/2881?journal_id=305572023-11-07T15:02:34ZBrandon Murphy
<ul></ul><p>I won't pretend to be a developer and tell you what function needs updating, I defer to you on that! :-)</p>
<p>Given the behavior mentioned in <a href="#note-28">#note-28</a>, I'm honestly surprised trailing spaces were not already normalized out, but I sure don't know enough to read through libhtp and suricata code to figure it out.</p>