https://redmine.openinfosecfoundation.org/https://redmine.openinfosecfoundation.org/favicon.ico?17011170022015-12-06T09:25:25ZOpen Information Security FoundationSuricata - Bug #1632: Fail to download large file with browser https://redmine.openinfosecfoundation.org/issues/1632?journal_id=57532015-12-06T09:25:25ZPeter Manevpetermanev@gmail.com
<ul></ul><p>Can you please describe/detail a bit more specific case scenario.</p>
<p>In the first part of your initial description - I am left with the impression that you have troubles downloading a large file with any browser or using wget - aka an end user download problem as opposed to file extraction. Is that correct or?</p> Suricata - Bug #1632: Fail to download large file with browser https://redmine.openinfosecfoundation.org/issues/1632?journal_id=57552015-12-06T09:58:07ZSamiux Arunnersam@gmail.com
<ul></ul><p>I do not encounter any download problem with wget even it also encounter timeout. However, I encounter download timeout with any browser even without md5 or filestore feature.</p> Suricata - Bug #1632: Fail to download large file with browser https://redmine.openinfosecfoundation.org/issues/1632?journal_id=57562015-12-07T03:07:56ZSamiux Arunnersam@gmail.com
<ul></ul><p>For example, I tried to download the following iso files with any browser, the download will be failed.</p>
<p><a class="external" href="http://cdimage.kali.org/kali-2.0/kali-linux-2.0-amd64.iso">http://cdimage.kali.org/kali-2.0/kali-linux-2.0-amd64.iso</a><br /><a class="external" href="https://www.backbox.org/downloads">https://www.backbox.org/downloads</a><br /><a class="external" href="https://www.microsoft.com/en-hk/software-download/windows10ISO">https://www.microsoft.com/en-hk/software-download/windows10ISO</a></p>
<p>However, Ubuntu iso can be downloaded with browser and without any problem.</p> Suricata - Bug #1632: Fail to download large file with browser https://redmine.openinfosecfoundation.org/issues/1632?journal_id=57582015-12-07T05:16:03ZVictor Julienvictor@inliniac.net
<ul></ul><p>Can you share your yaml?</p>
<p>It's interesting that you see this both with http and https links. That means that libhtp isn't a factor, as it's not used for https downloads.</p> Suricata - Bug #1632: Fail to download large file with browser https://redmine.openinfosecfoundation.org/issues/1632?journal_id=57602015-12-07T05:37:31ZSamiux Arunnersam@gmail.com
<ul></ul><p>My suricata.yaml is the below :</p>
<p><a class="external" href="http://pastebin.com/u2An2mmM">http://pastebin.com/u2An2mmM</a></p> Suricata - Bug #1632: Fail to download large file with browser https://redmine.openinfosecfoundation.org/issues/1632?journal_id=57652015-12-07T10:50:15ZSamiux Arunnersam@gmail.com
<ul><li><strong>File</strong> <a href="/attachments/1147">stats.log.tar.gz</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/1147/stats.log.tar.gz">stats.log.tar.gz</a> added</li></ul><p>I attached the stats.log for the downloading windows 10 iso.</p> Suricata - Bug #1632: Fail to download large file with browser https://redmine.openinfosecfoundation.org/issues/1632?journal_id=57662015-12-07T13:50:54ZSamiux Arunnersam@gmail.com
<ul><li><strong>File</strong> <a href="/attachments/1148">suricata.yaml</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/1148/suricata.yaml">suricata.yaml</a> added</li></ul><p>I did the following experiments :</p>
<p>Download Windows 10 iso file with Firefox (and add-ons enabled) on Suricata af_packet mode (IPS inline) -</p>
<p>(1) load without rules : the iso file download is completed and the speed is nearly my ISP connection speed.<br />(2) load with rules : the speed of the download will drop a lot and till 10kb to 0kb. The download then failed due to timeout.</p>
<p>I attached the suricata.yaml for reference.</p> Suricata - Bug #1632: Fail to download large file with browser https://redmine.openinfosecfoundation.org/issues/1632?journal_id=57772015-12-08T10:31:54ZVictor Julienvictor@inliniac.net
<ul><li><strong>Priority</strong> changed from <i>Urgent</i> to <i>Normal</i></li></ul><p>Are you able to narrow down which rules trigger this? It might be helpful to enable rule profiling, so you can see which rules are inspected (even if they don't fully match).</p> Suricata - Bug #1632: Fail to download large file with browser https://redmine.openinfosecfoundation.org/issues/1632?journal_id=57812015-12-08T21:36:23ZPeter Fyon
<ul></ul><p>I ran into this same issue yesterday. I compiled suricata 3.0RC2 from source to enable rule profiling and think I've narrowed it down. Here's the profiling that I did on basically just downloading a ubuntu iso (cancelled after about 100MB since the speed dropped to < 1% of my maximum bandwidth):</p>
<pre>
--------------------------------------------------------------------------
Date: 12/8/2015 -- 22:22:51
--------------------------------------------------------------------------
Num Rule Gid Rev Ticks % Checks Matches Max Ticks Avg Ticks Avg Match Avg No Match
-------- ------------ -------- -------- ------------ ------ -------- -------- ----------- ----------- ----------- --------------
1 2221000 1 1 656034977 13.40 60458 0 12811964 10851.09 0.00 10851.09
2 1000003 1 1 500851016 10.23 68496 0 12701488 7312.12 0.00 7312.12
3 2221002 1 1 203809237 4.16 19536 0 6602884 10432.50 0.00 10432.50
4 2020865 1 3 162494234 3.32 2792 0 3372952 58199.94 0.00 58199.94
5 2017552 1 6 115986768 2.37 19536 0 7618772 5937.08 0.00 5937.08
</pre>
<p>And the offending rules:<br />2221000<br />/etc/suricata/rules/http-events.rules:alert http any any -> any any (msg:"SURICATA HTTP unknown error"; flow:established; app-layer-event:http.unknown_error; flowint:http.anomaly.count,+,1; classtype:protocol-command-decode; sid:2221000; rev:1;)</p>
<p>1000003 (custom)<br />/etc/suricata/rules/local.rules:alert http any any -> any any (msg:"Flash video download"; filemagic:"Flash Video"; filestore; noalert; sid:1000003; rev:1;)</p>
<p>2221002<br />/etc/suricata/rules/http-events.rules:alert http any any -> any any (msg:"SURICATA HTTP request field missing colon"; flow:established,to_server; app-layer-event:http.request_field_missing_colon; flowint:http.anomaly.count,+,1; classtype:protocol-command-decode; sid:2221002; rev:1;)</p>
<p>2020865<br />/etc/suricata/rules/emerging-current_events.rules:alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"ET CURRENT_EVENTS Nuclear EK Landing Apr 08 2015"; flow:established,from_server; content:"Server|3a 20|nginx"; http_header; file_data; content:"Q|22|"; fast_pattern; content:"length"; pcre:"/^\s*?\<\s*?10/Rs"; content:"replace"; within:500; pcre:"/^\s*?\x28\s*?\x22\s\x22\s*?,\s*?\x22(?:\!(?:\x22\s*?\+\s*?\x22)?)?Q(?:\x22\s*?\+\s*?\x22)?Q\x22/Rs"; classtype:trojan-activity; sid:2020865; rev:3;)</p>
<p>2017552<br />/etc/suricata/rules/emerging-current_events.rules:alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET CURRENT_EVENTS Cushion Redirection"; flow:established,to_server; content:"/index.php?"; http_uri; content:"="; distance:1; within:1; http_uri; content:!"=aHR0"; http_uri; fast_pattern; pcre:"/\/index\.php\?[a-z]=[A-Za-z0-9\/\+]*?(?:(?:N1cm[kw]|RpbWU)9|(?:zdXJ[ps]|0aW1l)P|c3Vy(?:aT|bD)|dGltZT)[A-Za-z0-9\/\+]+?(?:(?:N1cm[kw]|RpbWU)9|(?:zdXJ[ps]|0aW1l)P|c3Vy(?:aT|bD)|dGltZT)[A-Za-z0-9\/\+]+(?:(?:N1cm[kw]|RpbWU)9|(?:zdXJ[ps]|0aW1l)P|c3Vy(?:aT|bD)|dGltZT)[A-Za-z0-9\/\+]+={0,2}$/U"; reference:url,malwaremustdie.blogspot.co.uk/2013/09/302-redirector-new-cushion-attempt-to.html; classtype:trojan-activity; sid:2017552; rev:6;)</p>
<p>Looks like those HTTP events are heavy.</p> Suricata - Bug #1632: Fail to download large file with browser https://redmine.openinfosecfoundation.org/issues/1632?journal_id=57862015-12-09T04:39:19ZVictor Julienvictor@inliniac.net
<ul></ul><p>Samiux, as a test, could you set these 2 libhtp settings to something much lower?</p>
<pre><code>request-body-limit: 0<br /> response-body-limit: 0</code></pre>
<p>e.g.<br /> request-body-limit: 1mb<br /> response-body-limit: 1mb</p>
<p>And see if that makes the issue go away?</p>
<p>Peter, do you have the same depth settings?</p> Suricata - Bug #1632: Fail to download large file with browser https://redmine.openinfosecfoundation.org/issues/1632?journal_id=57912015-12-09T09:37:30ZVictor Julienvictor@inliniac.net
<ul></ul><p>Could you test <a class="external" href="https://github.com/inliniac/suricata/pull/1788">https://github.com/inliniac/suricata/pull/1788</a>, think I found the issue.</p> Suricata - Bug #1632: Fail to download large file with browser https://redmine.openinfosecfoundation.org/issues/1632?journal_id=57932015-12-09T19:54:43ZPeter Fyon
<ul></ul><p>Hi Victor,</p>
<p>I spoke too soon and pointed the finger at those rules, but after commenting them out, the issue continued.</p>
<p>I just compiled that pull request and it looks like it's fixed the issue. Download speed hasn't dropped and my AFPacket thread isn't pegging the cpu anymore.</p> Suricata - Bug #1632: Fail to download large file with browser https://redmine.openinfosecfoundation.org/issues/1632?journal_id=57952015-12-10T04:13:09ZSamiux Arunnersam@gmail.com
<ul></ul><p>This patch only works for NFQueue IPS mode but not AF_PACKET IPS mode.</p> Suricata - Bug #1632: Fail to download large file with browser https://redmine.openinfosecfoundation.org/issues/1632?journal_id=57962015-12-10T04:16:02ZVictor Julienvictor@inliniac.net
<ul></ul><p>Can you expand on that Samiux? I tested with AF_PACKET IPS mode.</p> Suricata - Bug #1632: Fail to download large file with browser https://redmine.openinfosecfoundation.org/issues/1632?journal_id=58022015-12-10T14:36:52ZSamiux Arunnersam@gmail.com
<ul></ul><p>The patch fixes the problem. The AF_PACKET IPS mode problem is due to the ET POLICY rule. When the rule is disabled, the problem solved.</p> Suricata - Bug #1632: Fail to download large file with browser https://redmine.openinfosecfoundation.org/issues/1632?journal_id=58032015-12-10T20:01:42ZSamiux Arunnersam@gmail.com
<ul></ul><p>Sorry, I spoke too early that the problem was fixed.</p>
<p>Even I disabled sid 2000419 and 2018959, Windows 10 iso stalled at 285MB. There was no alert during download.</p>
<p>Suricata is running af-packet mode inline (IPS) with ET Open rules.</p> Suricata - Bug #1632: Fail to download large file with browser https://redmine.openinfosecfoundation.org/issues/1632?journal_id=58042015-12-11T03:52:19ZSamiux Arunnersam@gmail.com
<ul></ul><p>When download kali, it failed at 1.6gb.</p> Suricata - Bug #1632: Fail to download large file with browser https://redmine.openinfosecfoundation.org/issues/1632?journal_id=58092015-12-14T22:35:45ZSamiux Arunnersam@gmail.com
<ul></ul><p>I applied this pull request (<a class="external" href="https://github.com/inliniac/suricata/pull/1790">https://github.com/inliniac/suricata/pull/1790</a>) and download Windows 10 via Firefox from <a class="external" href="https://www.microsoft.com/en-us/software-download/techbench">https://www.microsoft.com/en-us/software-download/techbench</a> with the following setting at suricata.yaml and I get success result. Some false positive drop rules are disabled, for example ET WEB_CLIENT. The download is in almost in full speed of my internet connection (10/10Mb). Meanwhile, Kali Linux 2.0 can be downloaded with lower values at libhtp section.</p>
<pre><code>libhtp:</code></pre>
<pre><code>default-config:<br /> personality: IDS</code></pre>
<ol>
<li>Can be specified in kb, mb, gb. Just a number indicates</li>
<li>it's in bytes.<br /> request-body-limit: 10mb<br /> response-body-limit: 10mb</li>
</ol>
<ol>
<li>inspection limits<br /> request-body-minimal-inspect-size: 32kb<br /> request-body-inspect-window: 4kb<br /> response-body-minimal-inspect-size: 8192kb<br /> response-body-inspect-window: 1024kb</li>
</ol> Suricata - Bug #1632: Fail to download large file with browser https://redmine.openinfosecfoundation.org/issues/1632?journal_id=58132015-12-15T08:05:17ZVictor Julienvictor@inliniac.net
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Closed</i></li><li><strong>Assignee</strong> set to <i>Victor Julien</i></li><li><strong>Target version</strong> set to <i>3.0RC3</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li></ul>