Suricata

Here are some possible directions for reducing anomaly log activity:

• Rate limit log records. Use a mechanism like the Linux kernel's "printk ratelimit" that restricts the number of messages logged within a time interval. Log records that exceed the threshold are dropped; when drops occur, a log record is logged stating how many records were dropped when the next log record is written. The advantage of this approach is simplicity; the disadvantage is lost records..
• Store and forward. Batch successive log records into a fixed size memory area (size TBD). When memory area reaches capacity, the accumulated logs will be written. This maintains ordering at the expense of latency. An option would be to buffer messages until (1) a time threshold or (2) a size/count threshold is reached. Which ever occurs first, causes the log records to be written. This approach increases the memory footprint of Suricata but amortizes the cost to write over many records. This approach is simple, doesn't lose information, smooths jitter but uses more memory.
• Compress adjacent like records. Adjacent log records that are the same* (sameness tbd) would be accumulated with and marked with an occurrence value. This approach will store the last record that would've been logged and increase it's occurrence count as long as subsequent records are identical (TBD). When a non-identical record is submitted, the record being held is logged (output) and the non-identical record is buffered as long as subsequent records are identical. This is like the store and forward mechanism with store size of 1 and a semantic that identical records will be combined and the duplicates discarded. The chief disadvantage is complexity and performance may continue to suffer when few successive records are deemed identical.
• Filtering options. The chief drawback is that no relief may be provided when the filter choice doesn't match or isn't suitable for the anomalies that are occurring. Some ideas on filter choices:
• Filter on stream or packet events using the event code. Log records are packet events when the event code is less than or equal to DECODE_EVENT_PACKET_MAX.

• Filter on layer 3 protocol (unable to determine, ip, icmp)
• Filter on layer 4 protocol (udp, tcp, ...).
• Filter on layer 7 protocol (if available).
• Filter on whether packet is invalid (PKT_IS_INVALID) or not.
• Filter on specific decode events. This would be difficult to explain and configure.
• A combination of one or more of the preceding choices.

I think all or most suggestions from https://redmine.openinfosecfoundation.org/issues/2958#note-3 have been implemented. It would be nice to consider more advanced filters as well in this ticket.