Suricata

I'm not sure Suricata itself should output this, or it could be a done in a lua script based on the stats.

Victor Julien wrote:

I'm not sure Suricata itself should output this, or it could be a done in a lua script based on the stats.

Yes, it could be done somewhere else. However snort emits debug messages when memcap is exceeded, so it provides a good debugging tool right out of the box. Anyway it's not a priority.

I think it is very useful/helpful in terms of debug/tuning to have those (optionally enabled maybe) verbose/dbg msg in suricata.log with regards to memcaps being reached.

I think if we do such a thing we need a logging method that would allow us to say "log this once" or "log this no more than once a second". Any log message based on traffic is a risk wrt log flooding etc.

Victor Julien wrote:

I think if we do such a thing we need a logging method that would allow us to say "log this once" or "log this no more than once a second". Any log message based on traffic is a risk wrt log flooding etc.

Well, I think this can be either hardcoded or have a setting in a configuration file (like "memcap-limit-warn-count: 5", value of -1 will log it without any limit). And we can implement a macro like this one:

#define DO_FIRST_N(max, stmt) \
do { \
static volatile int logLimiter = 0;\
if (++logLimiter > (max)) \ { \
break; \
} \
stmt; \
} while (0);

And use it like:

DO_FIRST_N(1, SC_LOG_WARNING(...));

Maybe there's more neat and tidy solution :)

This variable logLimiter will only exist in the scope of the DO_FIRST_N(1, SC_LOG_WARNING(...)); statement, right? How will it be shared between threads or multiple invocations of a code block?

Victor Julien wrote:

This variable logLimiter will only exist in the scope of the DO_FIRST_N(1, SC_LOG_WARNING(...)); statement, right? How will it be shared between threads or multiple invocations of a code block?

If I remember correctly, static variables have a local scope but a global lifetime. So this one should be shared between threads and work correctly after multiple calls (however, there may be a non-critical race condition with this simple counter). Some time ago I did a quick test, with and without threads, and it seemed to work.
Maybe there's another solution for this, but I've tried to implement something like LOG_FIRST_N macro from google logging library (glog).