Project

General

Profile

Actions

Bug #1632

closed

Fail to download large file with browser

Added by Samiux A almost 7 years ago. Updated almost 7 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

I am using 3.0rc1 (github version). When using browser (no matter what browser is) to download a large size file (from about 100Mb to over 1Gb), the download will timeout and the download will fail in final. However, when using wget to download, even it encounters timeout, it can download with more tries and the download is completed.

I am running Suricata in af_packet mode and md5 as well as filestore are applied. I think the problem may be libhtp. This problem also happened in 2.1dev without md5 and filestore too. I have tested the problem many times in different network with the same problem.

Furthermore, I can download pdf files without any problem.


Files

stats.log.tar.gz (526 KB) stats.log.tar.gz Samiux A, 12/07/2015 10:49 AM
suricata.yaml (62.3 KB) suricata.yaml Samiux A, 12/07/2015 01:49 PM
Actions #1

Updated by Peter Manev almost 7 years ago

Can you please describe/detail a bit more specific case scenario.

In the first part of your initial description - I am left with the impression that you have troubles downloading a large file with any browser or using wget - aka an end user download problem as opposed to file extraction. Is that correct or?

Actions #2

Updated by Samiux A almost 7 years ago

I do not encounter any download problem with wget even it also encounter timeout. However, I encounter download timeout with any browser even without md5 or filestore feature.

Actions #3

Updated by Samiux A almost 7 years ago

For example, I tried to download the following iso files with any browser, the download will be failed.

http://cdimage.kali.org/kali-2.0/kali-linux-2.0-amd64.iso
https://www.backbox.org/downloads
https://www.microsoft.com/en-hk/software-download/windows10ISO

However, Ubuntu iso can be downloaded with browser and without any problem.

Actions #4

Updated by Victor Julien almost 7 years ago

Can you share your yaml?

It's interesting that you see this both with http and https links. That means that libhtp isn't a factor, as it's not used for https downloads.

Actions #5

Updated by Samiux A almost 7 years ago

My suricata.yaml is the below :

http://pastebin.com/u2An2mmM

Actions #6

Updated by Samiux A almost 7 years ago

I attached the stats.log for the downloading windows 10 iso.

Actions #7

Updated by Samiux A almost 7 years ago

I did the following experiments :

Download Windows 10 iso file with Firefox (and add-ons enabled) on Suricata af_packet mode (IPS inline) -

(1) load without rules : the iso file download is completed and the speed is nearly my ISP connection speed.
(2) load with rules : the speed of the download will drop a lot and till 10kb to 0kb. The download then failed due to timeout.

I attached the suricata.yaml for reference.

Actions #8

Updated by Victor Julien almost 7 years ago

  • Priority changed from Urgent to Normal

Are you able to narrow down which rules trigger this? It might be helpful to enable rule profiling, so you can see which rules are inspected (even if they don't fully match).

Actions #9

Updated by Peter Fyon almost 7 years ago

I ran into this same issue yesterday. I compiled suricata 3.0RC2 from source to enable rule profiling and think I've narrowed it down. Here's the profiling that I did on basically just downloading a ubuntu iso (cancelled after about 100MB since the speed dropped to < 1% of my maximum bandwidth):

 --------------------------------------------------------------------------
  Date: 12/8/2015 -- 22:22:51
  --------------------------------------------------------------------------
   Num      Rule         Gid      Rev      Ticks        %      Checks   Matches  Max Ticks   Avg Ticks   Avg Match   Avg No Match
  -------- ------------ -------- -------- ------------ ------ -------- -------- ----------- ----------- ----------- --------------
  1        2221000      1        1        656034977    13.40  60458    0        12811964    10851.09    0.00        10851.09
  2        1000003      1        1        500851016    10.23  68496    0        12701488    7312.12     0.00        7312.12
  3        2221002      1        1        203809237    4.16   19536    0        6602884     10432.50    0.00        10432.50
  4        2020865      1        3        162494234    3.32   2792     0        3372952     58199.94    0.00        58199.94
  5        2017552      1        6        115986768    2.37   19536    0        7618772     5937.08     0.00        5937.08

And the offending rules:
2221000
/etc/suricata/rules/http-events.rules:alert http any any -> any any (msg:"SURICATA HTTP unknown error"; flow:established; app-layer-event:http.unknown_error; flowint:http.anomaly.count,+,1; classtype:protocol-command-decode; sid:2221000; rev:1;)

1000003 (custom)
/etc/suricata/rules/local.rules:alert http any any -> any any (msg:"Flash video download"; filemagic:"Flash Video"; filestore; noalert; sid:1000003; rev:1;)

2221002
/etc/suricata/rules/http-events.rules:alert http any any -> any any (msg:"SURICATA HTTP request field missing colon"; flow:established,to_server; app-layer-event:http.request_field_missing_colon; flowint:http.anomaly.count,+,1; classtype:protocol-command-decode; sid:2221002; rev:1;)

2020865
/etc/suricata/rules/emerging-current_events.rules:alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"ET CURRENT_EVENTS Nuclear EK Landing Apr 08 2015"; flow:established,from_server; content:"Server|3a 20|nginx"; http_header; file_data; content:"Q|22|"; fast_pattern; content:"length"; pcre:"/^\s*?\<\s*?10/Rs"; content:"replace"; within:500; pcre:"/^\s*?\x28\s*?\x22\s\x22\s*?,\s*?\x22(?:\!(?:\x22\s*?\+\s*?\x22)?)?Q(?:\x22\s*?\+\s*?\x22)?Q\x22/Rs"; classtype:trojan-activity; sid:2020865; rev:3;)

2017552
/etc/suricata/rules/emerging-current_events.rules:alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET CURRENT_EVENTS Cushion Redirection"; flow:established,to_server; content:"/index.php?"; http_uri; content:"="; distance:1; within:1; http_uri; content:!"=aHR0"; http_uri; fast_pattern; pcre:"/\/index\.php\?[a-z]=[A-Za-z0-9\/\+]*?(?:(?:N1cm[kw]|RpbWU)9|(?:zdXJ[ps]|0aW1l)P|c3Vy(?:aT|bD)|dGltZT)[A-Za-z0-9\/\+]+?(?:(?:N1cm[kw]|RpbWU)9|(?:zdXJ[ps]|0aW1l)P|c3Vy(?:aT|bD)|dGltZT)[A-Za-z0-9\/\+]+(?:(?:N1cm[kw]|RpbWU)9|(?:zdXJ[ps]|0aW1l)P|c3Vy(?:aT|bD)|dGltZT)[A-Za-z0-9\/\+]+={0,2}$/U"; reference:url,malwaremustdie.blogspot.co.uk/2013/09/302-redirector-new-cushion-attempt-to.html; classtype:trojan-activity; sid:2017552; rev:6;)

Looks like those HTTP events are heavy.

Actions #10

Updated by Victor Julien almost 7 years ago

Samiux, as a test, could you set these 2 libhtp settings to something much lower?

request-body-limit: 0
response-body-limit: 0

e.g.
request-body-limit: 1mb
response-body-limit: 1mb

And see if that makes the issue go away?

Peter, do you have the same depth settings?

Actions #11

Updated by Victor Julien almost 7 years ago

Could you test https://github.com/inliniac/suricata/pull/1788, think I found the issue.

Actions #12

Updated by Peter Fyon almost 7 years ago

Hi Victor,

I spoke too soon and pointed the finger at those rules, but after commenting them out, the issue continued.

I just compiled that pull request and it looks like it's fixed the issue. Download speed hasn't dropped and my AFPacket thread isn't pegging the cpu anymore.

Actions #13

Updated by Samiux A almost 7 years ago

This patch only works for NFQueue IPS mode but not AF_PACKET IPS mode.

Actions #14

Updated by Victor Julien almost 7 years ago

Can you expand on that Samiux? I tested with AF_PACKET IPS mode.

Actions #15

Updated by Samiux A almost 7 years ago

The patch fixes the problem. The AF_PACKET IPS mode problem is due to the ET POLICY rule. When the rule is disabled, the problem solved.

Actions #16

Updated by Samiux A almost 7 years ago

Sorry, I spoke too early that the problem was fixed.

Even I disabled sid 2000419 and 2018959, Windows 10 iso stalled at 285MB. There was no alert during download.

Suricata is running af-packet mode inline (IPS) with ET Open rules.

Actions #17

Updated by Samiux A almost 7 years ago

When download kali, it failed at 1.6gb.

Actions #18

Updated by Samiux A almost 7 years ago

I applied this pull request (https://github.com/inliniac/suricata/pull/1790) and download Windows 10 via Firefox from https://www.microsoft.com/en-us/software-download/techbench with the following setting at suricata.yaml and I get success result. Some false positive drop rules are disabled, for example ET WEB_CLIENT. The download is in almost in full speed of my internet connection (10/10Mb). Meanwhile, Kali Linux 2.0 can be downloaded with lower values at libhtp section.

libhtp:
default-config:
personality: IDS
  1. Can be specified in kb, mb, gb. Just a number indicates
  2. it's in bytes.
    request-body-limit: 10mb
    response-body-limit: 10mb
  1. inspection limits
    request-body-minimal-inspect-size: 32kb
    request-body-inspect-window: 4kb
    response-body-minimal-inspect-size: 8192kb
    response-body-inspect-window: 1024kb
Actions #19

Updated by Victor Julien almost 7 years ago

  • Status changed from New to Closed
  • Assignee set to Victor Julien
  • Target version set to 3.0RC3
  • % Done changed from 0 to 100
Actions

Also available in: Atom PDF