Project

General

Profile

Actions
Copy link

Bug #1482

closed

suricata 2.1 beta4: StoreStateTxFileOnly crashes

Added by Alexander Gozman almost 10 years ago. Updated almost 10 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Victor Julien
Target version:
3.0RC1
Affected Versions:
Effort:
Difficulty:
Label:

Description

While testing with HTTP traffic with IXIA (10K concurrent HTTP connections) we've noticed that suricata crashes in StoreStateTxFileOnly. Suricata runs in af-packet inline mode on two interfaces. It works under load for some time and than gives:

suricata: detect-engine-state.c:387: StoreStateTxFileOnly: Assertion `!(1)' failed.

gdb shows the following:

Core was generated by `suricata -c /etc/suricata/suricata.yaml --af-packet'.
Program terminated with signal 6, Aborted.
#0 0x00007f97ce7e0625 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install suricata-2.1.b4-1.el6.x86_64
(gdb) bt
#0 0x00007f97ce7e0625 in raise () from /lib64/libc.so.6
#1 0x00007f97ce7e1e05 in abort () from /lib64/libc.so.6
#2 0x00007f97ce7d974e in __assert_fail_base () from /lib64/libc.so.6
#3 0x00007f97ce7d9810 in __assert_fail () from /lib64/libc.so.6
#4 0x00007f97d20aa90a in DeStateDetectStartDetection ()
#5 0x00007f97d2075edc in SigMatchSignatures ()
#6 0x00007f97d2076498 in Detect ()
#7 0x00007f97d2177868 in TmThreadsSlotVarRun ()
#8 0x00007f97d214e058 in AFPReadFromRing ()
#9 0x00007f97d214e5f1 in ReceiveAFPLoop ()
#10 0x00007f97d21780ce in TmThreadsSlotPktAcqLoop ()
#11 0x00007f97ced4d9d1 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f97ce8968fd in clone () from /lib64/libc.so.6

Is it a bug or it's caused with inappropriate settings for the given load? I've attached the YAML-file (a bit stripped and shuffled, 'cause it was processed with yaml-cpp library)

Files

suricata.yaml (8.3 KB) suricata.yaml Alexander Gozman, 06/09/2015 08:35 AM
Actions
Copy link
 #1

Updated by Alexander Gozman almost 10 years ago

Also, there was a truckload of signatures - about 28k, using 'ac' algorithm.

Actions
Copy link
 #2

Updated by Victor Julien almost 10 years ago

This means that AppLayerParserSetTxDetectState failed. Can you check what error code it failed with? Most likely would be -ENOMEM.

Actions
Copy link
 #3

Updated by Alexander Gozman almost 10 years ago

If I understood correctly, the issue may be caused with htp application layer - no-one else seems to return -ENOMEM... Also, htp layer uses HTPMalloc, which may return NULL if memcap is insufficient. I'll check this out, but if it's the case, probably, there should be an error message that describes a possible reason and how it may be fixed. This assertion failure is not self-explanatory ;)

Actions
Copy link
 #4

Updated by Alexander Gozman almost 10 years ago

Also, I'm not sure if coredumping with SIGABRT is a good way to handle this error :)

Actions
Copy link
 #5

Updated by Victor Julien almost 10 years ago

Yeah the BUG_ON should certainly be removed before 2.1 goes stable. I added it to make bugs/issues in this new code stand out, but it's probably time to remove it.

Actions
Copy link
 #6

Updated by Alexander Gozman almost 10 years ago

  • Status changed from New to Closed

Seems that it can be closed 'cause increasing http memcap solved the problem. Probably suricata should give a hint when memcap is exhausted.

Actions
Copy link
 #7

Updated by Peter Manev almost 10 years ago

Just to add a comment - could you please open a feature request for that -
a capability to use warning/hint in log format or similar when certain memcaps are reached -

that could be very helpful in verbose mode when doing troubleshooting.

Actions
Copy link
 #8

Updated by Victor Julien almost 10 years ago

This already exists, the http.memcap counter.

Actions
Copy link
 #9

Updated by Peter Manev almost 10 years ago

I meant it differently - the counter exists - but the warning that it is reached is what can be helpful in suricata.log for example. It could potentially be achieved through lua scripting as well - similar to your github surilua perf script - https://github.com/inliniac/surilua.

Actions
Copy link
 #10

Updated by Alexander Gozman almost 10 years ago

Peter Manev wrote:

Just to add a comment - could you please open a feature request for that -
a capability to use warning/hint in log format or similar when certain memcaps are reached -

that could be very helpful in verbose mode when doing troubleshooting.

Added low priority feature #1489.

Actions
Copy link
 #11

Updated by Victor Julien almost 10 years ago

  • Assignee set to Victor Julien
  • Priority changed from High to Normal
  • Target version set to 3.0RC1

I removed the BUG_ON.

Actions
Copy link

Also available in: Atom PDF