Suricata

So I noticed that Suricata was occasionally creating zero sized files in the file store, despite the related JSON saying it should be larger.

All these settings should be fine as from the looks of https://suricata.readthedocs.io/en/suricata-6.0.10/file-extraction/file-extraction.html it should not be placing a upper limit on size.

I've found I can repeatedly reproduce this issue via grabbing putty. That said dropped a copy at http://vvelox.net/test.exe to make testing easier via wgetting as I could not think of where else off hand there is a executable to test with that is easily available via http and not https as well as not redirecting to https.

Example JSON in which it created a zero sized file...

{
  "timestamp": "2023-02-16T20:41:58.380676+0000",
  "flow_id": 1064488670622519,
  "in_iface": "eth1",
  "event_type": "fileinfo",
  "src_ip": "98.102.84.2",
  "src_port": 80,
  "dest_ip": "10.112.39.45",
  "dest_port": 58856,
  "proto": "TCP",
  "http": {
    "hostname": "vvelox.net",
    "url": "/test.exe",
    "http_user_agent": "Wget/1.18 (linux-gnu)",
    "http_content_type": "application/x-msdownload",
    "http_method": "GET",
    "protocol": "HTTP/1.1",
    "status": 200,
    "length": 1647912
  },
  "app_proto": "http",
  "fileinfo": {
    "filename": "/test.exe",
    "sid": [
      2018959
    ],
    "magic": "PE32+ executable (GUI) x86-64, for MS Windows",
    "gaps": false,
    "state": "CLOSED",
    "md5": "f838fdafd0881cf1e6040a07d78e840d",
    "sha1": "2a35456b2f67bd12905378beb6eaf373f6a0d0d1",
    "sha256": "fc6f9dbdf4b9f8dd1f5f3a74cb6e55119d3fe2c9db52436e10ba07842e6c3d7c",
    "stored": true,
    "file_id": 243,
    "size": 1647912,
    "tx_id": 0
  }
}

Config bits for .stream...

memcap: 64mb
checksum-validation: no # reject wrong csums
inline: auto
reassembly:
  memcap: 256mb
  depth: 0
  toserver-chunk-size: 2560
  toclient-chunk-size: 2560
  randomize-chunk-size: yes

Config bits from .app-layer.protocols.http...

enabled: yes
libhtp:
  default-config:
    personality: IDS
    request-body-limit: 0
    response-body-limit: 0
    request-body-minimal-inspect-size: 32kb
    request-body-inspect-window: 4kb
    response-body-minimal-inspect-size: 40kb
    response-body-inspect-window: 16kb
    response-body-decompress-layer-limit: 2
    http-body-inline: auto
    swf-decompression:
      enabled: yes
      type: both
      compress-depth: 0
      decompress-depth: 0
    double-decode-path: no
    double-decode-query: no
  server-config:

Output section...

  - file-store:
      version: 2
      enabled: yes
      dir: /var/log/suricata/files
      write-fileinfo: yes
      stream-depth: 5242880
      force-hash: [sha1, md5]
      xff:
        enabled: no
        mode: extra-data
        deployment: reverse
        header: X-Forwarded-For

Output from suricata --build-info...

This is Suricata version 6.0.10 RELEASE
Features: PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LIBJANSSON TLS TLS_C11 MAGIC RUST
SIMD support: SSE_4_2 SSE_4_1 SSE_3
Atomic intrinsics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 6.3.0 20170516, C version 201112
compiled with -fstack-protector
compiled with _FORTIFY_SOURCE=2
L1 cache line size (CLS)=64
thread local storage method: _Thread_local
compiled with LibHTP v0.5.42, linked against LibHTP v0.5.42

Suricata Configuration:
  AF_PACKET support:                       yes
  eBPF support:                            no
  XDP support:                             no
  PF_RING support:                         no
  NFQueue support:                         no
  NFLOG support:                           no
  IPFW support:                            no
  Netmap support:                          no  using new api: no
  DAG enabled:                             no
  Napatech enabled:                        no
  WinDivert enabled:                       no

  Unix socket enabled:                     yes
  Detection enabled:                       yes

  Libmagic support:                        yes
  libnss support:                          yes
  libnspr support:                         yes
  libjansson support:                      yes
  hiredis support:                         yes
  hiredis async with libevent:             yes
  Prelude support:                         no
  PCRE jit:                                yes
  LUA support:                             no
  libluajit:                               no
  GeoIP2 support:                          no
  Non-bundled htp:                         no
  Hyperscan support:                       yes
  Libnet support:                          yes
  liblz4 support:                          yes
  HTTP2 decompression:                     no

  Rust support:                            yes
  Rust strict mode:                        no
  Rust compiler path:                      /usr/local/bin/rustc
  Rust compiler version:                   rustc 1.56.1 (59eed8a2a 2021-11-01)
  Cargo path:                              /usr/local/bin/cargo
  Cargo version:                           cargo 1.56.0 (4ed5d137b 2021-10-04)
  Cargo vendor:                            yes

  Python support:                          yes
  Python path:                             /usr/bin/python3
  Install suricatactl:                     yes
  Install suricatasc:                      yes
  Install suricata-update:                 yes

  Profiling enabled:                       no
  Profiling locks enabled:                 no

  Plugin support (experimental):           yes

Development settings:
  Coccinelle / spatch:                     no
  Unit tests enabled:                      no
  Debug output enabled:                    no
  Debug validation enabled:                no

Generic build parameters:
  Installation prefix:                     /usr
  Configuration directory:                 /etc/suricata/
  Log directory:                           /var/log/suricata/

  --prefix                                 /usr
  --sysconfdir                             /etc
  --localstatedir                          /var
  --datarootdir                            /usr/share

  Host:                                    x86_64-pc-linux-gnu
  Compiler:                                gcc (exec name) / g++ (real)
  GCC Protect enabled:                     yes
  GCC march native enabled:                yes
  GCC Profile enabled:                     yes
  Position Independent Executable enabled: no
  CFLAGS                                   -g -O2 -std=c11 -pg -march=native -I${srcdir}/../rust/gen -I${srcdir}/../rust/dist
  PCAP_CFLAGS                               -I/usr/include
  SECCFLAGS                                -fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security