Suricata

I would expect for this session, "state":"TRUNCATED".

This is really meant for increasing the depth, or disabling it (unlimited). If set to a non-zero value, it will only have affect if greater than your stream.reassembly.depth, as a certain amount of depth is required for other things to function properly.

Did you have a requirement where you only wish a portion of a file to be logged, even if more of it is in memory?

pascal delalande wrote:

Using suricata 4.1 beta2 (top of git - May 30 2018), and http session with this yaml:
- file-store:
enabled: yes # set to yes to enable
log-dir: files # directory to store the files
force-magic: no # force logging magic on all stored files
force-hash: [md5]
force-filestore: yes # force storing of all files
stream-depth: 1
whatever size of attached files within HTTP session, stored files are never truncated despite stream-depth is set at a fixed small value.
Example: stream-depth set to 1, we still have meta file like:

more file.8.meta
TIME: 05/30/2018-21:58:02.767116
SRC IP: 192.168.0.26
DST IP: 212.95.74.42
PROTO: 6
SRC PORT: 54303
DST PORT: 80
APP PROTO: http
HTTP URI: /async/articles/flashs
HTTP HOST: www.lefigaro.fr
HTTP REFERER: http://www.lefigaro.fr/
HTTP USER AGENT: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0
FILENAME: /async/articles/flashs
MAGIC: <unknown>
STATE: CLOSED
SIZE: 55545

{"timestamp":"2018-05-30T21:58:02.848356+0200",....."fileinfo":{"filename":"/async/articles/flashs","gaps":false,"state":"CLOSED","stored":true,"file_id":8,"size":55545,"tx_id":0}}

I do think this is not the expected behavior.

pascal delalande wrote:

Using suricata 4.1 beta2 (top of git - May 30 2018), and http session with this yaml:
- file-store:
enabled: yes # set to yes to enable
log-dir: files # directory to store the files
force-magic: no # force logging magic on all stored files
force-hash: [md5]
force-filestore: yes # force storing of all files
stream-depth: 1
whatever size of attached files within HTTP session, stored files are never truncated despite stream-depth is set at a fixed small value.
Example: stream-depth set to 1, we still have meta file like:

more file.8.meta
TIME: 05/30/2018-21:58:02.767116
SRC IP: 192.168.0.26
DST IP: 212.95.74.42
PROTO: 6
SRC PORT: 54303
DST PORT: 80
APP PROTO: http
HTTP URI: /async/articles/flashs
HTTP HOST: www.lefigaro.fr
HTTP REFERER: http://www.lefigaro.fr/
HTTP USER AGENT: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0
FILENAME: /async/articles/flashs
MAGIC: <unknown>
STATE: CLOSED
SIZE: 55545

{"timestamp":"2018-05-30T21:58:02.848356+0200","flow_id":1136624478607845,"in_iface":"wlan0","event_type":"fileinfo",../articles/flashs","gaps":false,"state":"CLOSED","stored":true,"file_id":8,"size":55545,"tx_id":0}}

I do think this is not the expected behavior.

pascal delalande wrote:

Using suricata 4.1 beta2 (top of git - May 30 2018), and http session with this yaml:
- file-store:
enabled: yes # set to yes to enable
log-dir: files # directory to store the files
force-magic: no # force logging magic on all stored files
force-hash: [md5]
force-filestore: yes # force storing of all files
stream-depth: 1
whatever size of attached files within HTTP session, stored files are never truncated despite stream-depth is set at a fixed small value.
Example: stream-depth set to 1, we still have meta file like:

more file.8.meta
TIME: 05/30/2018-21:58:02.767116
SRC IP: 192.168.0.26
DST IP: 212.95.74.42
PROTO: 6
SRC PORT: 54303
DST PORT: 80
APP PROTO: http
HTTP URI: /async/articles/flashs
HTTP HOST: www.lefigaro.fr
HTTP REFERER: http://www.lefigaro.fr/
HTTP USER AGENT: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0
FILENAME: /async/articles/flashs
MAGIC: <unknown>
STATE: CLOSED
SIZE: 55545

I do think this is not the expected behavior.

Sorry, I don't think any new comment came through.

I'm not sure if this is really an issue, per my original comment.

If the reassembly buffer is larger than the filestore stream depth, we have more data that we can log, so we do it. The filestore stream-depth doesn't reduce the other stream stream-depth values from stream and libhtp. Perhaps when the filestore stream-depth is less than the data available, it acts as a file size limit for logging? The name doesn't quite make sense then.

However, when this value is greater than the stream.reassembly.depth, then it does override, in which case the name does make sense.

All-in-all, the multiple places to set stream-depth are a little confusing.

Agreed. I think its a good idea to consider the various 'stream depth' settings as overrides to the master stream.reassembly.depth setting for specific protocols or operations like file storing.

I think we need to consider whether it makes sense to make this a 'increase only' override (e.g. main depth is 1MiB, file store override is 10MiB) or that we want to allow decrease as well (e.g. main depth unlimited but modbus only 10MiB). It's definitely created with the 'increase only' usecases in mind.

It's clear we need a bit of value validation as well. A setting of 1 from the example is nonsensical. A minimum value of at least a few kilobyes seems to make sense.

I think if we agree on the 'increase only' we should also validate that the overrides are in fact higher than the main setting and ignore them with a warning if they are not.

My recommendation, at least for filestore is that it is a grow only option. I don't think a filestore configuration should negatively impact stream reassembly for http inspection.

So a notice log if the file store stream depth is set less than primary stream depth.

I still wonder though if this should then be applied as a max file size? Even if the stream depth is 1mb, but the file-store.stream-depth is 500k for example, should the file extraction just stop at 500k?

Jason Ish wrote:

I still wonder though if this should then be applied as a max file size? Even if the stream depth is 1mb, but the file-store.stream-depth is 500k for example, should the file extraction just stop at 500k?

No, there is no 1 to 1 relation between stream depth and file size. HTTP bodies may have chunking, multipart, compression etc all affecting the size of the files that are transferred. It should simply remain a setting that affects the stream engines depth logic.

I think we need to consider whether it makes sense to make this a 'increase only' override (e.g. main depth is 1MiB, file store override is 10MiB) or that we want to allow decrease as well (e.g. main depth unlimited but modbus only 10MiB). It's definitely created with the 'increase only' usecases in mind.

The default value for the stream reassembly depth originates in

suricata.yaml.in

Jeff Lucovsky wrote:

I think we need to consider whether it makes sense to make this a 'increase only' override (e.g. main depth is 1MiB, file store override is 10MiB) or that we want to allow decrease as well (e.g. main depth unlimited but modbus only 10MiB). It's definitely created with the 'increase only' usecases in mind.

The default value for the stream reassembly depth originates in [...]. Where is the corresponding default value for the file store override located?

There isn't one. This value only kicks in if present, and acts as an override.

Might be nice to get this cleaned up for 5.0.0 as it looks like trivial code changes, and some documentation clarification.