document content inspection in chunks
Suricata is not always alerting on traffic that matches content in rules.
When testing Suricata on my virtual machine (Virtual Box 5.2.6) running CentOS Linux release 7.4.1708 (Core), I found there were some situations when traffic was not triggering alerts but was still being picked up in the http.log. As best as I can tell, the behavior is influenced by the combination of the location of the content in the packets and also whether or not the content is split between packets , though it cannot reliably be reproduced. For my tests I was using nc (Ncat version 6.40 http://nmap.org/ncat) to serve the content on the VM where Suricata is hosted, and then cURL on a separate physical machine to retrieve it from the VM with Suricata. There is NAT'ing and a port forward to the VM running Suricata in this case.
It is possible there is something weird going on in the way the traffic is passed to the VM so that Suricata is behaving as expected, but I wanted to report this issue anyway especially after reading Bug #2427.
I was able to reproduce this on Suricata 3.x versions and also 4.x versions (4.0.3, 4.0.4, 4.1.0-beta1). It is worth noting that the 3.x versions had some differences in what traffic was picked up when compared to 4.x versions. I was also more reliably able to reproduce the behavior on the 3.x versions, meaning when content was in a specific location it always did NOT trigger an alert. For the sake of the issue here I plan to focus on the 4.x versions.
The rules file is attached as custom.rules. I tried to make my rules as simple as possible so that there was not much room for error.
I ran some packet captures on the VM where Suricata is located. These should allow for reliably reproducing the behavior I have seen. The attachment pcaps.tar.gz contains these captures. I tried to name the captures to describe the expected results. There is even one capture where the same payload is served three times yet only one of the three streams triggers an alert. Here are some details on the captures:
This capture does NOT trigger an alert in my tests but it seems should trigger the "catch-me" rule. Content is split between frames 19 and 20.
This capture does trigger an alert. The content "catch-me" is contained in frame 20.
This capture has 3 streams and should match the rule with the "hit" content. The first 2 streams do NOT trigger alerts and the last one does. It is the same payload served in all three cases.
This capture does NOT trigger an alert. The "hit" content is split between frames 11 and 12.
Steps to reproduce:
1. Use vanilla suricata.yaml config. Set HOME_NET to appropriate address and update rule-files section to point to the attached custom.rules file. (Tested with both ac and hs mpm-algos and reproducible in both cases)
2. Run Suricata in pcap file mode with attached packet captures using "suricata -r <pcap name>"
The traffic in the captures does not always trigger alerts even though the rules seem to be configured in a way that should match the traffic. Every stream is logged in the http.log file.
The traffic in the captures should trigger for every stream since they each contain content that match in custom.rules.
Updated by Andreas Herz over 3 years ago
- Status changed from New to Feedback
- Assignee changed from OISF Dev to Eric Urban
Eric, I was able to reproduce this issue and also "fix" it. You need to increase the chunk size settings:
stream: memcap: 64mb checksum-validation: yes # reject wrong csums inline: auto # auto will use inline mode in IPS mode, yes or no set it statically reassembly: memcap: 256mb depth: 1mb # reassemble 1mb into a stream toserver-chunk-size: 25600 toclient-chunk-size: 25600 randomize-chunk-size: yes
Updated by Eric Urban about 3 years ago
Sorry, I must have missed the notification from this issue as I just saw it now when logging into Redmine.
Thank you, Andreas, for pointing me toward the config option to affect this behavior! I tested adjusting this option and there are now alerts in all cases. I looked these settings up in the docs and found on https://suricata.readthedocs.io/en/suricata-4.1.5/configuration/suricata-yaml.html#stream-engine that there is the text "To avoid making the borders predictable, the sizes van be varied by adding in a random factor." I believe this applies to the conversation here since and if I am understanding this right if this randomize-chunk-size option didn't exist then there would be more reason to worry about missed detections.
Updated by Brian Jones about 3 years ago
Looking at the c file, the underscores need to be changed to hyphens in the documentation. Andreas example is correct, the doc/userguide/configuration/suricata-yaml.rst example is not. I can't submit a pull request at the moment. I will next week if someone else doesn't before then.
Inspection of reassembled data is done in chunks. The size of these chunks is set with -->toserver_chunk_size<-- and -->toclient_chunk_size<--.
memcap: 256mb # Memory reserved for stream data reconstruction (in bytes)
depth: 1mb # The depth of the reassembling.
-->toserver_chunk_size<--: 2560 # inspect raw stream in chunks of at least this size
-->toclient_chunk_size<--: 2560 # inspect raw stream in chunks of at least