Bug #1016: Suricata Only Captures And Stores The First ~4900 bytes of a file. - Suricata - Open Information Security Foundation

Actions

Copy link

Bug #1016

closed

Suricata Only Captures And Stores The First ~4900 bytes of a file.

Added by Adam Bradbury over 10 years ago. Updated over 8 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Target version:

Affected Versions:

Effort:

Difficulty:

Label:

Description

Hi there,

When ever Suricata tries to extract a file from live network traffic it only extracts roughly the first 4900 bytes, leaving me with corrupt files.

Here are the details of my setup:

I have the Suricata 2,0beta1 release

This is Suricata version 2.0beta1 RELEASE

I have changed my libhtp config:

libhtp:
         default-config:
           personality: IDS
           request-body-limit: 0
           response-body-limit: 0
           request-body-minimal-inspect-size: 32kb
           request-body-inspect-window: 8kb
           response-body-minimal-inspect-size: 32kb
           response-body-inspect-window: 8kb

And the stream config:

stream:
  memcap: 128mb
  checksum-validation: no
  inline: auto                  
  reassembly:
    memcap: 64mb
    depth: 32mb                  
    toserver-chunk-size: 2560
    toclient-chunk-size: 2560
    randomize-chunk-size: yes@

Here is my file capture config:

- file-store:
      enabled: yes       
      log-dir: files    
      force-magic: no   
      force-md5: no

- file-log:
      enabled: yes
      filename: files-json.log
      append: yes
      force-magic: no   
      force-md5: no

I'm running a single rule as per the wiki:

alert http any any -> any any (msg:"FILE pdf detected"; filemagic:"PDF document"; filestore; sid:3; rev:1;)

The alert fires correctly when I download a pdf. But the file I get as "file.1" is only around ~4900 bytes long every time, all the meta information is correct and the pdf downloads correctly to my workstation. I have tried this with flow:established,to_client, with the same result. I have also tried it with image files and get the same result and the same file sizes.

I have followed all the tutorials, what do I have to do to fix this or is this an issue that has been fixed in later releases?

Actions

Copy link

Updated by Peter Manev over 10 years ago

apt-get install ethtool

Example ->
INFO on the card:


@suricata:~$ sudo ethtool -k eth3
Offload parameters for eth3:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: on
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on

NOTICE small k and LARGE K !!!
To change :

pedro@suricata:~$ sudo ethtool -K eth3 tso off
pedro@suricata:~$ sudo ethtool -K eth3 gso off
pedro@suricata:~$ sudo ethtool -K eth3 gro off
pedro@suricata:~$ sudo ethtool -K eth3 ufo off
pedro@suricata:~$ sudo ethtool -K eth3 tx off
pedro@suricata:~$ sudo ethtool -K eth3 rx off

Everything should be OFF in order not to impact file extraction.
Please note that these changes are not persistant after a reboot.

Actions

Copy link

Updated by Adam Bradbury over 10 years ago

Ran the ethtool commands and still not working:

[root\@ids files]# ethtool -k em1
  Features for em1: 
  rx-checksumming: off
  tx-checksumming: off
  scatter-gather: off
  tcp-segmentation-offload: off
  udp-fragmentation-offload: off
  generic-segmentation-offload: off
  generic-receive-offload: off
  large-receive-offload: off
  rx-vlan-offload: on
  tx-vlan-offload: on
  ntuple-filters: off
  receive-hashing: off

Files folder still showing file sizes around 4900

[root@ids files]# ls lr
   total 24
    -rw-r--r-. 1 root root  470 Oct 29 13:08 file.2.meta
    rw-r--r-. 1 root root 4879 Oct 29 13:08 file.2
    rw-r--r-. 1 root root  675 Oct 29 13:08 file.1.meta
    rw-r--r-. 1 root root 4928 Oct 29 13:08 file.1

Any ideas, i'm completely stumped?

Actions

Copy link

Updated by Peter Manev over 10 years ago

Can you try to change
inline: auto
to
inline: no

and see if any difference?

Actions

Copy link

Updated by Adam Bradbury over 10 years ago

Changed, restarted and still no difference.

It's as if it reads the first packet and disregards everything else that is left in the stream.

When i capture pictures it saves the first row of pixels from the top of the photo but nothing else, so its only grabbing the first packet I think.

Actions

Copy link

Updated by Peter Manev over 10 years ago

Clear the cashe of your browser (if you are using one for the tests) and try to downlaod different pdfs.
Do not use httpS :) links, if you are using one.

Actions

Copy link

Updated by Adam Bradbury over 10 years ago

All the PDF's I have downloaded have been different and from standard HTTP servers.

Actions

Copy link

Updated by Peter Manev over 10 years ago

Are you running on a Virtual machine?
If yes, what hypervisor are you using? Some hypervisors have offloading enabled by default and there is no way to see/change that.

Actions

Copy link