Suricata

Also, there was a truckload of signatures - about 28k, using 'ac' algorithm.

This means that AppLayerParserSetTxDetectState failed. Can you check what error code it failed with? Most likely would be -ENOMEM.

If I understood correctly, the issue may be caused with htp application layer - no-one else seems to return -ENOMEM... Also, htp layer uses HTPMalloc, which may return NULL if memcap is insufficient. I'll check this out, but if it's the case, probably, there should be an error message that describes a possible reason and how it may be fixed. This assertion failure is not self-explanatory ;)

Also, I'm not sure if coredumping with SIGABRT is a good way to handle this error :)

Yeah the BUG_ON should certainly be removed before 2.1 goes stable. I added it to make bugs/issues in this new code stand out, but it's probably time to remove it.

Just to add a comment - could you please open a feature request for that -
a capability to use warning/hint in log format or similar when certain memcaps are reached -

that could be very helpful in verbose mode when doing troubleshooting.

This already exists, the http.memcap counter.

I meant it differently - the counter exists - but the warning that it is reached is what can be helpful in suricata.log for example. It could potentially be achieved through lua scripting as well - similar to your github surilua perf script - https://github.com/inliniac/surilua.

Peter Manev wrote:

Just to add a comment - could you please open a feature request for that -
a capability to use warning/hint in log format or similar when certain memcaps are reached -

that could be very helpful in verbose mode when doing troubleshooting.

Added low priority feature #1489.