Project

General

Profile

Actions
Copy link

Support #2369

closed

option force-filestore generate truncated file

Added by Nicolas Danjon over 6 years ago. Updated about 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Affected Versions:
Label:

Description

Version used : Suricata 4.0.3

The option "force-filestore: yes" works using a 3.2.X version.
Using the pcap file at the following address : https://home.regit.org/~regit/slides.pcap :

1)

Command used :
/usr/local/bin/suricata -c /etc/suricata/suricata.yaml -r /home/pickle_rick/IDS/slides.pcap -v -k none

File-store part of the configuration file suricata.yaml:

 - file-store:
      enabled: yes       # set to yes to enable
      log-dir: /home/pickle_rick/IDS/    # directory to store the files
      force-magic: no   # force logging magic on all stored files
      # force logging of checksums, available hash functions are md5,
      # sha1 and sha256
      #force-hash: [md5]
      force-filestore: yes # force storing of all files
      # override global stream-depth for sessions in which we want to
      # perform file extraction. Set to 0 for unlimited.
      stream-depth: 0
      #waldo: file.waldo # waldo file to store the file_id across runs
      # uncomment to disable meta file writing
      write-meta: yes
      # uncomment the following variable to define how many files can
      # remain open for filestore by Suricata. Default value is 0 which
      # means files get closed after each write
      #max-open-files: 1000

Logs :

18/12/2017 -- 11:34:53 - <Notice> - This is Suricata version 4.0.3 RELEASE
18/12/2017 -- 11:34:53 - <Info> - CPUs/cores online: 4
18/12/2017 -- 11:34:57 - <Info> - 38 rule files processed. 13043 rules successfully loaded, 0 rules failed
18/12/2017 -- 11:34:57 - <Info> - Threshold config parsed: 0 rule(s) found
18/12/2017 -- 11:34:57 - <Info> - 13048 signatures processed. 1129 are IP-only rules, 5538 are inspecting packet payload, 8026 inspect application layer, 0 are decoder event only
18/12/2017 -- 11:34:58 - <Info> - fast output device (regular) initialized: fast.log
18/12/2017 -- 11:34:58 - <Info> - eve-log output device (regular) initialized: eve.json
18/12/2017 -- 11:34:58 - <Info> - stats output device (regular) initialized: stats.log
18/12/2017 -- 11:34:58 - <Info> - forcing filestore of all files
18/12/2017 -- 11:34:58 - <Info> - storing files in /home/pickle_rick/IDS/
18/12/2017 -- 11:34:58 - <Info> - reading pcap file /home/pickle_rick/IDS/slides.pcap
18/12/2017 -- 11:34:58 - <Notice> - all 5 packet processing threads, 4 management threads initialized, engine started.
18/12/2017 -- 11:34:58 - <Info> - No packets with invalid checksum, assuming checksum offloading is NOT used
18/12/2017 -- 11:34:58 - <Info> - pcap file end of file reached (pcap err code 0)
18/12/2017 -- 11:34:58 - <Notice> - Signal Received.  Stopping engine.
18/12/2017 -- 11:34:58 - <Info> - time elapsed 0.081s
18/12/2017 -- 11:34:58 - <Notice> - Pcap-file module read 2240 packets, 1974173 bytes
18/12/2017 -- 11:34:58 - <Info> - (W#01) Files extracted 0
18/12/2017 -- 11:34:58 - <Info> - (W#02) Files extracted 0
18/12/2017 -- 11:34:58 - <Info> - (W#03) Files extracted 1
18/12/2017 -- 11:34:58 - <Info> - (W#04) Files extracted 0
18/12/2017 -- 11:34:58 - <Info> - Alerts: 0
18/12/2017 -- 11:34:58 - <Info> - cleaning up signature grouping structure... complete

Trying to open the extracted file, we get the following error:
PDF document is damaged

Size of the extracted file :
1023K déc. 18 11:34 file.1

file.1.meta content :

TIME:              02/09/2015-08:02:33.754535
PCAP PKT NUM:      47
SRC IP:            2001:41d0:0001:9598:0000:0000:0000:0001
DST IP:            2a01:0e35:1394:5bd0:da50:e6ff:fe3c:3250
PROTO:             6
SRC PORT:          80
DST PORT:          59525
APP PROTO:         http
HTTP URI:          /~regit/ids-suricata-esiea.pdf
HTTP HOST:         home.regit.org
HTTP REFERER:      <unknown>
HTTP USER AGENT:   Wget/1.16 (linux-gnu)
FILENAME:          /~regit/ids-suricata-esiea.pdf
MAGIC:             <unknown>
STATE:             UNKNOWN
SIZE:              1046656

2)

Command used :
/usr/local/bin/suricata -c /etc/suricata/suricata.yaml -r /home/pickle_rick/IDS/slides.pcap -v -k none -S /etc/suricata/rules/my_pdf.rules

File-store part of the configuration file suricata.yaml:

- file-store:
      enabled: yes       # set to yes to enable
      log-dir: /home/pickle_rick/IDS/    # directory to store the files
      force-magic: no   # force logging magic on all stored files
      # force logging of checksums, available hash functions are md5,
      # sha1 and sha256
      #force-hash: [md5]
      force-filestore: no # force storing of all files
      # override global stream-depth for sessions in which we want to
      # perform file extraction. Set to 0 for unlimited.
      stream-depth: 0
      #waldo: file.waldo # waldo file to store the file_id across runs
      # uncomment to disable meta file writing
      write-meta: yes
      # uncomment the following variable to define how many files can
      # remain open for filestore by Suricata. Default value is 0 which
      # means files get closed after each write
      #max-open-files: 1000

Rule my_pdf.rules :

alert http any any -> any any (msg:"FILE PDF file claimed"; fileext:"pdf"; filestore; sid:1; rev:1;)
alert http any any -> any any (msg:"FILE pdf detected"; filemagic:"PDF document"; filestore; sid:2; rev:1;)

Logs :

18/12/2017 -- 11:40:37 - <Notice> - This is Suricata version 4.0.3 RELEASE
18/12/2017 -- 11:40:37 - <Info> - CPUs/cores online: 4
18/12/2017 -- 11:40:37 - <Info> - 1 rule files processed. 2 rules successfully loaded, 0 rules failed
18/12/2017 -- 11:40:37 - <Info> - Threshold config parsed: 0 rule(s) found
18/12/2017 -- 11:40:37 - <Info> - 2 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 2 inspect application layer, 0 are decoder event only
18/12/2017 -- 11:40:37 - <Info> - fast output device (regular) initialized: fast.log
18/12/2017 -- 11:40:37 - <Info> - eve-log output device (regular) initialized: eve.json
18/12/2017 -- 11:40:37 - <Info> - stats output device (regular) initialized: stats.log
18/12/2017 -- 11:40:37 - <Info> - storing files in /home/pickle_rick/IDS/
18/12/2017 -- 11:40:37 - <Info> - reading pcap file /home/pickle_rick/IDS/slides.pcap
18/12/2017 -- 11:40:37 - <Notice> - all 5 packet processing threads, 4 management threads initialized, engine started.
18/12/2017 -- 11:40:37 - <Info> - No packets with invalid checksum, assuming checksum offloading is NOT used
18/12/2017 -- 11:40:37 - <Info> - pcap file end of file reached (pcap err code 0)
18/12/2017 -- 11:40:37 - <Notice> - Signal Received.  Stopping engine.
18/12/2017 -- 11:40:37 - <Info> - time elapsed 0.040s
18/12/2017 -- 11:40:37 - <Notice> - Pcap-file module read 2240 packets, 1974173 bytes
18/12/2017 -- 11:40:37 - <Info> - (W#01) Files extracted 0
18/12/2017 -- 11:40:37 - <Info> - (W#02) Files extracted 0
18/12/2017 -- 11:40:37 - <Info> - (W#03) Files extracted 1
18/12/2017 -- 11:40:37 - <Info> - (W#04) Files extracted 0
18/12/2017 -- 11:40:37 - <Info> - Alerts: 2
18/12/2017 -- 11:40:37 - <Info> - cleaning up signature grouping structure... complete

The extracted file has the right size and can be opened without any error.

Size of the extracted file:
1,7M déc. 18 11:40 file.1

file.1.meta content :

TIME:              02/09/2015-08:02:33.754535
PCAP PKT NUM:      47
SRC IP:            2001:41d0:0001:9598:0000:0000:0000:0001
DST IP:            2a01:0e35:1394:5bd0:da50:e6ff:fe3c:3250
PROTO:             6
SRC PORT:          80
DST PORT:          59525
APP PROTO:         http
HTTP URI:          /~regit/ids-suricata-esiea.pdf
HTTP HOST:         home.regit.org
HTTP REFERER:      <unknown>
HTTP USER AGENT:   Wget/1.16 (linux-gnu)
FILENAME:          /~regit/ids-suricata-esiea.pdf
MAGIC:             PDF document, version 1.5
STATE:             CLOSED
SIZE:              1740729

Files

Screenshot from 2020-03-25 06-51-11.png (12.6 KB) Screenshot from 2020-03-25 06-51-11.png Victor Julien, 03/25/2020 05:51 AM

Related issues 3 (0 open — 3 closed)

Related to Suricata - Bug #2495: Stream depth and filestore interactionClosedActions
Related to Suricata - Bug #2264: file-store.stream-depth not working as expected when configured to a specfic valueClosedGiuseppe LongoActions
Related to Suricata - Bug #2506: filestore v1: with stream-depth not null, files are never truncated ClosedJeff LucovskyActions
Actions
Copy link
 #1

Updated by Andreas Herz over 6 years ago

  • Assignee set to OISF Dev
  • Target version set to TBD
Actions
Copy link
 #2

Updated by Andreas Herz almost 5 years ago

I can confirm that this is still an issue with 5.0 beta, might be related to #2495, #2264 and #2506

Actions
Copy link
 #3

Updated by Victor Julien almost 5 years ago

  • Related to Bug #2495: Stream depth and filestore interaction added
Actions
Copy link
 #4

Updated by Victor Julien almost 5 years ago

  • Related to Bug #2264: file-store.stream-depth not working as expected when configured to a specfic value added
Actions
Copy link
 #5

Updated by Victor Julien almost 5 years ago

  • Related to Bug #2506: filestore v1: with stream-depth not null, files are never truncated added
Actions
Copy link
 #6

Updated by Victor Julien about 4 years ago

The issue is simply that there are a few data segments missing from the stream, leading to a GAP. So reporting TRUNCATED is correct.

Actions
Copy link

Also available in: Atom PDF