Project

General

Profile

Bug #3703

fileinfo "stored: false" even if the file is kept on disk

Added by Armature Technologies 10 months ago. Updated about 24 hours ago.

Status:
In Progress
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:
Needs backport to 5.0, Needs backport to 6.0

Description

Hi,

We believe we found a reproducible bug in the file store code.
Basically, what happens is that some streams contain files that are correctly extracted, dumped on disk, marked as closed (not truncated), and moved from tmp/ to the filestore while the fileinfo eve-log indicates that the file was not stored. We have been observing this for a while now, and were recently able to reproduce it in lab, on Suricata 4.1.8 and Suricata 5.0.3.

This is problematic, especially if one is doing automated analysis of the extracted files and not using `write-fileinfo: yes`, because it means that the file might remain "unknown" and never be automatically analyzed.

The suricata config is nothing fancy; we kept the default config from the Ubuntu package, with file-store v2 enabled, and a very basic filemagic rule:

```
alert http any any -> any any (msg: "FILEMAGIC PDF document"; filemagic:"PDF document"; filestore:both,file; noalert; sid:1100008; rev:1;)
```

The bug can be reproduced with the attached pcap file. This is not an isolated case; we were able to reproduce it with several other streams that have similar flow profile. Specifically, after the initial handshake, the file is pushed in its entirety, without any acknowledgment, then all acknowledgment segments are sent and the connection is closed abruptly with a RST. While this way of doing TCP is dirty efficient, it remains valid.

Please reach out to us if you need any more details regarding this issue.

Thank you.

Florian Maury for ArmatureTech DevTeam


Files

pdf_http_467659.pcap (12.3 KB) pdf_http_467659.pcap Armature Technologies, 05/18/2020 01:12 PM

Related issues

Copied to Bug #4382: fileinfo "stored: false" even if the file is kept on diskAssignedJeff LucovskyActions
Copied to Bug #4383: fileinfo "stored: false" even if the file is kept on diskAssignedShivani BhardwajActions
#1

Updated by Armature Technologies 15 days ago

Hi,

Just an quick update from our side: we have been able to confirm that the issue is still present in the latest stable version of Suricata (6.0.1 at the time of writing). The fileinfo event continues to indicate that the file was not stored even though it effectively was stored.

We have heard feedback from production networks where these false reports represent north of 25% of files stored by Suricata. This problem is significant and we are a bit abashed that this bug report has remained unanswered and unaddressed, as accuracy is probably one of the most desirable attributes of an IDS.

We have not been able to come-up with a fix of our own for this issue, and help would be very much appreciated.

Thank you.

Cheers,
Florian Maury

#2

Updated by Victor Julien 1 day ago

  • Status changed from New to Assigned
  • Assignee set to Jeff Lucovsky
  • Target version set to 7.0beta1
  • Label Needs backport to 5.0, Needs backport to 6.0 added
#3

Updated by Jeff Lucovsky about 24 hours ago

  • Status changed from Assigned to In Progress

I'm not able to reproduce on either

master
nor
master-5.0.x

With this rule:

alert http any any -> any any (msg: "FILEMAGIC PDF document"; filemagic:"PDF document"; filestore:both,file; noalert; sid:1100008; rev:1;)

This event is generated

$ jq -r  'select(.event_type=="fileinfo")' < /tmp/ll/eve.json
{
  "timestamp": "2019-09-30T05:16:05.585275-0400",
  "flow_id": 1031263655160533,
  "pcap_cnt": 17,
  "event_type": "fileinfo",
  "src_ip": "192.168.0.2",
  "src_port": 80,
  "dest_ip": "192.168.0.1",
  "dest_port": 36656,
  "proto": "TCP",
  "http": {
    "hostname": "192.168.0.2",
    "url": "/467659.pdf",
    "http_content_type": "application/pdf",
    "http_method": "GET",
    "protocol": "HTTP/1.1",
    "status": 200,
    "length": 9952
  },
  "app_proto": "http",
  "fileinfo": {
    "filename": "/467659.pdf",
    "sid": [],
    "gaps": false,
    "state": "CLOSED",
    "sha256": "02f43016d07812f881dc1ccee724f95682016ff00c7ee6b2c856d4d693ce3fa5",
    "stored": true,
    "file_id": 1,
    "size": 9952,
    "tx_id": 0
  }
}

#4

Updated by Jeff Lucovsky about 24 hours ago

  • Copied to Bug #4382: fileinfo "stored: false" even if the file is kept on disk added
#5

Updated by Jeff Lucovsky about 24 hours ago

  • Copied to Bug #4383: fileinfo "stored: false" even if the file is kept on disk added

Also available in: Atom PDF