Suricata

My first suggestion would be to look at packet capture issues such as bad csums, vlan issue or async routing. There are some hints at https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Self_Help_Diagrams

Can you add a stats.log record here?

Hey, thanks for the TS link! I went through the no-alerts flow: I enabled almost all of the emerging-threats (8/8/2018 -- 15:54:25 - <Info> - 37 rule files processed. 16971 rules successfully loaded, 0 rules failed), and there are alerts that happen but they are all of a very few categories. Feeding fast.log into tail shows that there are alerts for STUN, P2P, and ET DNS queries, but that's it; I haven't looked at all of the included emerging threats rules in detail, but I assumed that there are some including http attack traffic which we see very regularly (Against our platforms, as well as outgoing from students).

I included the last update to stats.log below; I have seen the occasional invalid checksum reported in there, but it's along the lines of two packets in billions and billions.

Let me know if there is any other helpful information I can provide!

stats.log

------------------------------------------------------------------------------------
Date: 8/8/2018 -- 16:05:18 (uptime: 0d, 00h 03m 17s)
------------------------------------------------------------------------------------
Counter                                    | TM Name                   | Value
------------------------------------------------------------------------------------
capture.kernel_packets                     | Total                     | 12511568
capture.kernel_drops                       | Total                     | 1054769
decoder.pkts                               | Total                     | 11478971
decoder.bytes                              | Total                     | 8849140969
decoder.ipv4                               | Total                     | 10629773
decoder.ipv6                               | Total                     | 849206
decoder.ethernet                           | Total                     | 11478971
decoder.tcp                                | Total                     | 10546793
decoder.udp                                | Total                     | 928459
decoder.icmpv4                             | Total                     | 1533
decoder.icmpv6                             | Total                     | 38
decoder.teredo                             | Total                     | 15
decoder.avg_pkt_size                       | Total                     | 770
decoder.max_pkt_size                       | Total                     | 1514
flow.tcp                                   | Total                     | 114457
flow.udp                                   | Total                     | 46640
flow.icmpv6                                | Total                     | 9
tcp.sessions                               | Total                     | 76946
tcp.syn                                    | Total                     | 79717
tcp.synack                                 | Total                     | 1428
tcp.rst                                    | Total                     | 33112
detect.alert                               | Total                     | 915
app_layer.flow.dcerpc_udp                  | Total                     | 1
app_layer.flow.dns_udp                     | Total                     | 31071
app_layer.tx.dns_udp                       | Total                     | 52807
app_layer.flow.failed_udp                  | Total                     | 15568
flow_mgr.new_pruned                        | Total                     | 102433
flow.spare                                 | Total                     | 10728
flow_mgr.flows_checked                     | Total                     | 5189
flow_mgr.flows_notimeout                   | Total                     | 4456
flow_mgr.flows_timeout                     | Total                     | 733
flow_mgr.flows_removed                     | Total                     | 733
flow_mgr.rows_checked                      | Total                     | 65536
flow_mgr.rows_skipped                      | Total                     | 62378
flow_mgr.rows_empty                        | Total                     | 275
flow_mgr.rows_maxlen                       | Total                     | 7
tcp.memuse                                 | Total                     | 22937600
tcp.reassembly_memuse                      | Total                     | 3276800
dns.memuse                                 | Total                     | 2238817
flow.memuse                                | Total                     | 24183520

The stats are interesting. There is quite a bit of packet loss. This will impact detection.

Also, the ratio of syn vs syn/ack, is very skewed. This could be due to SYN scans/floods, but it might also be an indicator that you're not capturing all traffic. Is there async routing involved? If so, enabling the 'stream.async-oneside' option could help, although the best results reached when the full traffic is seen by Suricata.

I would focus on these 2 things first, and see if improving them improves the alerts as well.

Thanks Victor,

I enabled stream.async-oneside in the config, and the behaviour seems to continue (The alert never triggers, at all. It still triggers if I just use content: "POST" several times a second, and some captured traffic shows posts happening). We do have async routing, we have four internet links and likely not enough hardware to parse 40G of traffic (Even if we could get this all to one location, it would still be split across multiple links).

I did find out that it seems the lua script (Which uses log()) is called even if there is not an alert to log? (When I had the single signature with http_method) I'm not clear on the details of that; I thought only an alert would trip that. I changed the luascript to needs["type"] = "packet" and did some changes trying to use the raw packet data; I'm seeing what look like posts but it is a lot harder without all of the helper functions. OTOH, I have confirmed that it is seeing data with HTTP posts on the regular, in raw packet form using SCPacketPayload()

Do things work correctly when you disable the lua script? Do you experience packet loss then?

I disabled the lua script, and re-enabled some of the emerging threat rules in suricata.yaml. Re-running it (With my very simple custom signature loaded on the command line) resulted in some detection from the emerging threats rules (policy privacy violations, at a glance of fast.log), no hits on the rule I had added (Which should be triggering on any outbound POST).

I also enabled http.log, but it only creates a 0 byte log file.

During use without the emerging-threats rules files enabled, I routinely see packet drop numbers that are like 0.01%, so I feel that any packet loss here could just be a red herring; The system likely would drop a lot of packets trying to alert on all of the rules, but that's not the intended purpose; I only want it to alert (And call the lua function) on one very specific but seemingly very simple rule. The stats.log I posted is also only 8% or so packet loss, which would not be good for an IDS but still doesn't seem to explain why an alert on outbound POST isn't working? The most recent run with just some of the emerging-threats rules files enabled reported a 0.34% packet loss on the command line when I ctrl+C'd to stop it, and 0 invalid checksums

My next step in debugging this would be to capture samples of traffic using tcpdump or a similar capture tool and inspect it manually to see if you really capture the HTTP traffic you're trying to match on.

Thanks Victor,

In the original issue I mentioned that I had used Suricatas pcap mode briefly to generate a pcap of the traffic on that interface: I used it on purpose to eliminate any problems with a different capture method. I downloaded the resultant pcap and opened it in wireshark, and quickly found multiple http POST transactions that were leaving the network, as an http stream I could follow (As I understand, the stream and the sizes meant it was over multiple packets, so I assumed if wireshark could assemble them, suricata ashould be able to? This obviously may be incorrect). Is there anything in particular else that I should be looking for in that pcap? I could fire up tcpdump or something, although I'm concerned that the traffic may be faster than it can write to disk, I've never tested that in a high throughput environment. Is there a difference to using that instead of capturing from suricata?

Suricata doesn't have an easy way to quickly capture some traffic from the commandline, like tcpdump does. Suricata's capture is more meant to be used together with all regular operations the engine does. As those regular ops aren't working very well, it is probably easier to just use tcpdump for now. Use a bpf to limit what you capture. E.g. "tcpdump -w output.pcap -s0 -i <interface> tcp port 80"

Hey, sorry for the delay here, I was pulled away to some other work.

I still have the original pcap that I captured using suricata; it loads into wireshark on my local machine without any problems, and it has multiple HTTP post streams that can be followed. I also tried with https://scapy.net/ in Python (As well as an addin to add http as an understood layer) and was able to trivially create something using that, on the tapped interface, that triggered for POST requests going across that interface. It triggers multiple times a minute, as expected; I'd wanted to use suricata for this because of performance and multi-threading concerns, but I was investigating this as a possible alternative (The single threaded performance trying to parse 10G of traffic is unsurprisingly a problem).

So, the pcap from suricata, as well as other tools (scapy wraps libpcap, as I understand it, and I've also used tshark with BPF filters and some simple python wrapping on the XML output to confirm it also sees POST traffic), all confirm that multiple HTTP POST requests are flowing across the interface regularly, and it seems that the very simple rule should be triggered as a result? Also, everything works on the copper interface so I'm still assuming this has to be either a configuration problem I'm not aware of, or some strange edge-case bug?

You mention it works on copper but not over the SFP port - is it possible that this is due to NIC settings? For example your offloading is set correctly and such ?

Hi,

I believe so? I mean, it works fine with multiple other products, and I've tried a lot of settings in suricata; Is there something in particular to look at here? Something that would cause suricata to not trigger on this traffic, but others to see it?

Apologies for the late reply.
If it works on copper but not on fiber it is most likely something with the NICs settings - probably offloading and maybe queues.