Bug #1482
closedsuricata 2.1 beta4: StoreStateTxFileOnly crashes
Description
While testing with HTTP traffic with IXIA (10K concurrent HTTP connections) we've noticed that suricata crashes in StoreStateTxFileOnly. Suricata runs in af-packet inline mode on two interfaces. It works under load for some time and than gives:
suricata: detect-engine-state.c:387: StoreStateTxFileOnly: Assertion `!(1)' failed.
gdb shows the following:
Core was generated by `suricata -c /etc/suricata/suricata.yaml --af-packet'.
Program terminated with signal 6, Aborted.
#0 0x00007f97ce7e0625 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install suricata-2.1.b4-1.el6.x86_64
(gdb) bt
#0 0x00007f97ce7e0625 in raise () from /lib64/libc.so.6
#1 0x00007f97ce7e1e05 in abort () from /lib64/libc.so.6
#2 0x00007f97ce7d974e in __assert_fail_base () from /lib64/libc.so.6
#3 0x00007f97ce7d9810 in __assert_fail () from /lib64/libc.so.6
#4 0x00007f97d20aa90a in DeStateDetectStartDetection ()
#5 0x00007f97d2075edc in SigMatchSignatures ()
#6 0x00007f97d2076498 in Detect ()
#7 0x00007f97d2177868 in TmThreadsSlotVarRun ()
#8 0x00007f97d214e058 in AFPReadFromRing ()
#9 0x00007f97d214e5f1 in ReceiveAFPLoop ()
#10 0x00007f97d21780ce in TmThreadsSlotPktAcqLoop ()
#11 0x00007f97ced4d9d1 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f97ce8968fd in clone () from /lib64/libc.so.6
Is it a bug or it's caused with inappropriate settings for the given load? I've attached the YAML-file (a bit stripped and shuffled, 'cause it was processed with yaml-cpp library)
Files
Updated by Alexander Gozman almost 10 years ago
Also, there was a truckload of signatures - about 28k, using 'ac' algorithm.
Updated by Victor Julien almost 10 years ago
This means that AppLayerParserSetTxDetectState failed. Can you check what error code it failed with? Most likely would be -ENOMEM.
Updated by Alexander Gozman almost 10 years ago
If I understood correctly, the issue may be caused with htp application layer - no-one else seems to return -ENOMEM... Also, htp layer uses HTPMalloc, which may return NULL if memcap is insufficient. I'll check this out, but if it's the case, probably, there should be an error message that describes a possible reason and how it may be fixed. This assertion failure is not self-explanatory ;)
Updated by Alexander Gozman almost 10 years ago
Also, I'm not sure if coredumping with SIGABRT is a good way to handle this error :)
Updated by Victor Julien almost 10 years ago
Yeah the BUG_ON should certainly be removed before 2.1 goes stable. I added it to make bugs/issues in this new code stand out, but it's probably time to remove it.
Updated by Alexander Gozman over 9 years ago
- Status changed from New to Closed
Seems that it can be closed 'cause increasing http memcap solved the problem. Probably suricata should give a hint when memcap is exhausted.
Updated by Peter Manev over 9 years ago
Just to add a comment - could you please open a feature request for that -
a capability to use warning/hint in log format or similar when certain memcaps are reached -
that could be very helpful in verbose mode when doing troubleshooting.
Updated by Victor Julien over 9 years ago
This already exists, the http.memcap counter.
Updated by Peter Manev over 9 years ago
I meant it differently - the counter exists - but the warning that it is reached is what can be helpful in suricata.log for example. It could potentially be achieved through lua scripting as well - similar to your github surilua perf script - https://github.com/inliniac/surilua.
Updated by Alexander Gozman over 9 years ago
Peter Manev wrote:
Just to add a comment - could you please open a feature request for that -
a capability to use warning/hint in log format or similar when certain memcaps are reached -that could be very helpful in verbose mode when doing troubleshooting.
Added low priority feature #1489.
Updated by Victor Julien over 9 years ago
- Assignee set to Victor Julien
- Priority changed from High to Normal
- Target version set to 3.0RC1
I removed the BUG_ON.