Project

General

Profile

Actions
Copy link

Bug #1016

closed
AB

Suricata Only Captures And Stores The First ~4900 bytes of a file.

Bug #1016: Suricata Only Captures And Stores The First ~4900 bytes of a file.

Added by Adam Bradbury almost 13 years ago. Updated over 10 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
Affected Versions:
Effort:
Difficulty:
Label:

Description

Hi there,

When ever Suricata tries to extract a file from live network traffic it only extracts roughly the first 4900 bytes, leaving me with corrupt files.

Here are the details of my setup:

I have the Suricata 2,0beta1 release

This is Suricata version 2.0beta1 RELEASE

I have changed my libhtp config:

libhtp:
         default-config:
           personality: IDS
           request-body-limit: 0
           response-body-limit: 0
           request-body-minimal-inspect-size: 32kb
           request-body-inspect-window: 8kb
           response-body-minimal-inspect-size: 32kb
           response-body-inspect-window: 8kb

And the stream config:

stream:
  memcap: 128mb
  checksum-validation: no
  inline: auto                  
  reassembly:
    memcap: 64mb
    depth: 32mb                  
    toserver-chunk-size: 2560
    toclient-chunk-size: 2560
    randomize-chunk-size: yes@

Here is my file capture config:

- file-store:
      enabled: yes       
      log-dir: files    
      force-magic: no   
      force-md5: no    
- file-log:
      enabled: yes
      filename: files-json.log
      append: yes
      force-magic: no   
      force-md5: no

I'm running a single rule as per the wiki:

alert http any any -> any any (msg:"FILE pdf detected"; filemagic:"PDF document"; filestore; sid:3; rev:1;)

The alert fires correctly when I download a pdf. But the file I get as "file.1" is only around ~4900 bytes long every time, all the meta information is correct and the pdf downloads correctly to my workstation. I have tried this with flow:established,to_client, with the same result. I have also tried it with image files and get the same result and the same file sizes.

I have followed all the tutorials, what do I have to do to fix this or is this an issue that has been fixed in later releases?

PM Updated by Peter Manev almost 13 years ago Actions
Copy link
 #1 

apt-get install ethtool

Example ->
INFO on the card:


@suricata:~$ sudo ethtool -k eth3
Offload parameters for eth3:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: on
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on

NOTICE small k and LARGE K !!!
To change :

pedro@suricata:~$ sudo ethtool -K eth3 tso off
pedro@suricata:~$ sudo ethtool -K eth3 gso off
pedro@suricata:~$ sudo ethtool -K eth3 gro off
pedro@suricata:~$ sudo ethtool -K eth3 ufo off
pedro@suricata:~$ sudo ethtool -K eth3 tx off
pedro@suricata:~$ sudo ethtool -K eth3 rx off

Everything should be OFF in order not to impact file extraction.
Please note that these changes are not persistant after a reboot.

AB Updated by Adam Bradbury almost 13 years ago Actions
Copy link
 #2

Ran the ethtool commands and still not working:

[root\@ids files]# ethtool -k em1
  Features for em1: 
  rx-checksumming: off
  tx-checksumming: off
  scatter-gather: off
  tcp-segmentation-offload: off
  udp-fragmentation-offload: off
  generic-segmentation-offload: off
  generic-receive-offload: off
  large-receive-offload: off
  rx-vlan-offload: on
  tx-vlan-offload: on
  ntuple-filters: off
  receive-hashing: off

Files folder still showing file sizes around 4900

[root@ids files]# ls lr
   total 24
    -rw-r--r-. 1 root root  470 Oct 29 13:08 file.2.meta
    rw-r--r-. 1 root root 4879 Oct 29 13:08 file.2
    rw-r--r-. 1 root root  675 Oct 29 13:08 file.1.meta
    rw-r--r-. 1 root root 4928 Oct 29 13:08 file.1

Any ideas, i'm completely stumped?

PM Updated by Peter Manev almost 13 years ago Actions
Copy link
 #3

Can you try to change
inline: auto
to
inline: no

and see if any difference?

AB Updated by Adam Bradbury almost 13 years ago Actions
Copy link
 #4

Changed, restarted and still no difference.

It's as if it reads the first packet and disregards everything else that is left in the stream.

When i capture pictures it saves the first row of pixels from the top of the photo but nothing else, so its only grabbing the first packet I think.

PM Updated by Peter Manev almost 13 years ago Actions
Copy link
 #5

Clear the cashe of your browser (if you are using one for the tests) and try to downlaod different pdfs.
Do not use httpS :) links, if you are using one.

AB Updated by Adam Bradbury almost 13 years ago Actions
Copy link
 #6

All the PDF's I have downloaded have been different and from standard HTTP servers.

PM Updated by Peter Manev almost 13 years ago Actions
Copy link
 #7

Are you running on a Virtual machine?
If yes, what hypervisor are you using? Some hypervisors have offloading enabled by default and there is no way to see/change that.

AB Updated by Adam Bradbury almost 13 years ago Actions
Copy link
 #8

No its not running on a VM, its on a Centos 5.6 box I have been using to run Snort on.

PM Updated by Peter Manev almost 13 years ago Actions
Copy link
 #9

I can not reproduce your issue.
But I am running Ubuntu and 3.2 kernel. I have not tried to reproduce it on CentOS 5.6.

Can you share your suricata.yaml , kernel level - privately if you would like?

VJ Updated by Victor Julien almost 13 years ago Actions
Copy link
 #10

  • Target version deleted (2.0beta1)

"Target version" is meant for planning the version an issue will be addressed in.

AH Updated by Andreas Herz over 10 years ago Actions
Copy link
 #11

  • Target version set to TBD

VJ Updated by Victor Julien over 10 years ago Actions
Copy link
 #12

  • Status changed from New to Closed
  • Target version deleted (TBD)
Actions
Copy link

Also available in: PDF Atom