Bug #706
closeddetect.alert - not incrementing in daemon mode
Added by Peter Manev over 13 years ago. Updated about 13 years ago.
Description
On our teams test box - even though we receive a big number of alerts,
the detect.alert counter in stats.log is not incrementing.
Files
| Bug706.txt (65.7 KB) Bug706.txt | Peter Manev, 02/24/2013 01:45 PM |
VJ Updated by Victor Julien over 13 years ago Actions #1
Basic info is missing: version (rev), runmode, etc.
PM Updated by Peter Manev over 13 years ago Actions #2
Please find the basic info needed below:
suricata --build-info [8174] 14/1/2013 -- 14:59:20 - (suricata.c:560) <Info> (SCPrintBuildInfo) -- This is Suricata version 1.4dev (rev 5f4c528) [8174] 14/1/2013 -- 14:59:20 - (suricata.c:633) <Info> (SCPrintBuildInfo) -- Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 PF_RING AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK HAVE_HTP_TX_GET_RESPONSE_HEADERS_RAW HAVE_NSS PROFILING [8174] 14/1/2013 -- 14:59:20 - (suricata.c:647) <Info> (SCPrintBuildInfo) -- 64-bits, Little-endian architecture [8174] 14/1/2013 -- 14:59:20 - (suricata.c:649) <Info> (SCPrintBuildInfo) -- GCC version 4.6.3, C version 199901 [8174] 14/1/2013 -- 14:59:20 - (suricata.c:655) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1 [8174] 14/1/2013 -- 14:59:20 - (suricata.c:658) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2 [8174] 14/1/2013 -- 14:59:20 - (suricata.c:661) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4 [8174] 14/1/2013 -- 14:59:20 - (suricata.c:664) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8 [8174] 14/1/2013 -- 14:59:20 - (suricata.c:667) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_16 [8174] 14/1/2013 -- 14:59:20 - (suricata.c:671) <Info> (SCPrintBuildInfo) -- compiled with -fstack-protector [8174] 14/1/2013 -- 14:59:20 - (suricata.c:677) <Info> (SCPrintBuildInfo) -- compiled with _FORTIFY_SOURCE=2 [8174] 14/1/2013 -- 14:59:20 - (suricata.c:680) <Info> (SCPrintBuildInfo) -- compiled with libhtp 0.2.11, linked against 0.2.11
using af_packet/workers
anything else that might be helpful?
VJ Updated by Victor Julien over 13 years ago Actions #3
Are other counters being updated normally?
PM Updated by Peter Manev over 13 years ago Actions #4
funny you say that...
actually no - it does not look like all of them are incremented - some are, but most are not (Ex):
capture.kernel_packets | AFPacketeth316 | 0
capture.kernel_drops | AFPacketeth316 | 0
decoder.pkts | AFPacketeth316 | 0
decoder.bytes | AFPacketeth316 | 0
decoder.ipv4 | AFPacketeth316 | 0
decoder.ipv6 | AFPacketeth316 | 0
decoder.ethernet | AFPacketeth316 | 0
decoder.raw | AFPacketeth316 | 0
decoder.sll | AFPacketeth316 | 0
decoder.tcp | AFPacketeth316 | 0
decoder.udp | AFPacketeth316 | 0
decoder.sctp | AFPacketeth316 | 0
decoder.icmpv4 | AFPacketeth316 | 0
decoder.icmpv6 | AFPacketeth316 | 0
decoder.ppp | AFPacketeth316 | 0
decoder.pppoe | AFPacketeth316 | 0
decoder.gre | AFPacketeth316 | 0
decoder.vlan | AFPacketeth316 | 0
decoder.teredo | AFPacketeth316 | 0
decoder.ipv4_in_ipv6 | AFPacketeth316 | 0
decoder.ipv6_in_ipv6 | AFPacketeth316 | 0
decoder.avg_pkt_size | AFPacketeth316 | 0
decoder.max_pkt_size | AFPacketeth316 | 0
defrag.ipv4.fragments | AFPacketeth316 | 0
defrag.ipv4.reassembled | AFPacketeth316 | 0
defrag.ipv4.timeouts | AFPacketeth316 | 0
defrag.ipv6.fragments | AFPacketeth316 | 0
defrag.ipv6.reassembled | AFPacketeth316 | 0
defrag.ipv6.timeouts | AFPacketeth316 | 0
defrag.max_frag_hits | AFPacketeth316 | 0
tcp.sessions | AFPacketeth316 | 0
tcp.ssn_memcap_drop | AFPacketeth316 | 0
tcp.pseudo | AFPacketeth316 | 0
tcp.invalid_checksum | AFPacketeth316 | 0
tcp.no_flow | AFPacketeth316 | 0
tcp.reused_ssn | AFPacketeth316 | 0
tcp.memuse | AFPacketeth316 | 0
tcp.syn | AFPacketeth316 | 0
tcp.synack | AFPacketeth316 | 0
tcp.rst | AFPacketeth316 | 0
tcp.segment_memcap_drop | AFPacketeth316 | 0
tcp.stream_depth_reached | AFPacketeth316 | 0
tcp.reassembly_memuse | AFPacketeth316 | 0
tcp.reassembly_gap | AFPacketeth316 | 0
detect.alert | AFPacketeth316 | 0
flow_mgr.closed_pruned | FlowManagerThread | 717373974
flow_mgr.new_pruned | FlowManagerThread | 1260698120
flow_mgr.est_pruned | FlowManagerThread | 216749232
flow.memuse | FlowManagerThread | 362540720
flow.spare | FlowManagerThread | 1089093
flow.emerg_mode_entered | FlowManagerThread | 0
flow.emerg_mode_over | FlowManagerThread | 0
thanks
VJ Updated by Victor Julien over 13 years ago Actions #5
@Eric could we be staying in AFPReadFromRing all the time? We need to call SCPerfSyncCountersIfSignalled(tv, 0); regularly for counters to work. It is called from ReceiveAFPLoop, but maybe we never leave AFPReadFromRing?
@Peter, are the counters updated when you stop suri?
PM Updated by Peter Manev over 13 years ago Actions #6
@Peter, are the counters updated when you stop suri?
no, they are not.
PM Updated by Peter Manev over 13 years ago Actions #7
Here the issue is similar to
https://redmine.openinfosecfoundation.org/issues/714
in -D mode it does not write counters info in stats.log - but in regular mode (non daemon) it updates the stats.log no problem.
root@suricata:/var/data/regit/log/suricata# suricata --build-info [23104] 19/1/2013 -- 13:01:43 - (suricata.c:560) <Info> (SCPrintBuildInfo) -- This is Suricata version 1.4dev (rev 5f4c528) [23104] 19/1/2013 -- 13:01:43 - (suricata.c:633) <Info> (SCPrintBuildInfo) -- Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 PF_RING AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK HAVE_HTP_TX_GET_RESPONSE_HEADERS_RAW HAVE_NSS PROFILING [23104] 19/1/2013 -- 13:01:43 - (suricata.c:647) <Info> (SCPrintBuildInfo) -- 64-bits, Little-endian architecture [23104] 19/1/2013 -- 13:01:43 - (suricata.c:649) <Info> (SCPrintBuildInfo) -- GCC version 4.6.3, C version 199901 [23104] 19/1/2013 -- 13:01:43 - (suricata.c:655) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1 [23104] 19/1/2013 -- 13:01:43 - (suricata.c:658) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2 [23104] 19/1/2013 -- 13:01:43 - (suricata.c:661) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4 [23104] 19/1/2013 -- 13:01:43 - (suricata.c:664) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8 [23104] 19/1/2013 -- 13:01:43 - (suricata.c:667) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_16 [23104] 19/1/2013 -- 13:01:43 - (suricata.c:671) <Info> (SCPrintBuildInfo) -- compiled with -fstack-protector [23104] 19/1/2013 -- 13:01:43 - (suricata.c:677) <Info> (SCPrintBuildInfo) -- compiled with _FORTIFY_SOURCE=2 [23104] 19/1/2013 -- 13:01:43 - (suricata.c:680) <Info> (SCPrintBuildInfo) -- compiled with libhtp 0.2.11, linked against 0.2.11
VJ Updated by Victor Julien over 13 years ago Actions #8
- Subject changed from detect.alert - not incrementing to detect.alert - not incrementing in daemon mode
- Assignee set to OISF Dev
- Target version set to 1.4.1
Is 1.3.5 also affected?
VJ Updated by Victor Julien about 13 years ago Actions #9
Peter, how did you start suricata?
I noticed the following: when testing like this: "suricata -i eth0 -c suricata.yaml -S local.rules -D" it didn't work. "detect.alert" not incrementing, fast.log empty.
When I looked at Suricata's output however, I noticed what causes it. local.rules was not found. When not in daemon mode, the CWD is where ever you start suri, in daemon mode the CWD is /. So local.rules was loaded from /local.rules, and it wasn't there.
When I started like this "suricata -c suricata.yaml -i eth0 -S /full/path/to/local.rules -D" it works. Both detect.alert and fast.log are fine in this case.
PM Updated by Peter Manev about 13 years ago Actions #10
suricata -c /etc/suricata.yaml --af-packet=eth3 -D
is how i started it.
VJ Updated by Victor Julien about 13 years ago Actions #11
Hmm, then I can't reproduce the issue. AF_PACKET + workers, in daemon mode:
detect.alert | AFPacketeth01 | 0 detect.alert | AFPacketeth02 | 0 detect.alert | AFPacketeth03 | 1 detect.alert | AFPacketeth04 | 0 detect.alert | AFPacketeth05 | 1 detect.alert | AFPacketeth06 | 0 detect.alert | AFPacketeth07 | 1 detect.alert | AFPacketeth08 | 1 detect.alert | AFPacketeth09 | 1 detect.alert | AFPacketeth010 | 0 detect.alert | AFPacketeth011 | 0 detect.alert | AFPacketeth012 | 0 detect.alert | AFPacketeth013 | 0 detect.alert | AFPacketeth014 | 0 detect.alert | AFPacketeth015 | 1 detect.alert | AFPacketeth016 | 0 detect.alert | AFPacketeth017 | 0 detect.alert | AFPacketeth018 | 0 detect.alert | AFPacketeth019 | 0 detect.alert | AFPacketeth020 | 0 detect.alert | AFPacketeth021 | 0 detect.alert | AFPacketeth022 | 0 detect.alert | AFPacketeth023 | 1 detect.alert | AFPacketeth024 | 0 detect.alert | AFPacketeth025 | 1 detect.alert | AFPacketeth026 | 0 detect.alert | AFPacketeth027 | 2 detect.alert | AFPacketeth028 | 1 detect.alert | AFPacketeth029 | 0 detect.alert | AFPacketeth030 | 0 detect.alert | AFPacketeth031 | 0 detect.alert | AFPacketeth032 | 0
Can you send over your yaml?
PM Updated by Peter Manev about 13 years ago Actions #12
- File Bug706.txt Bug706.txt added
shared the yaml.
I also attached some more info - after running for about 8:45 min ... all counters(except 4) are 0..
which is puzzling.
EL Updated by Eric Leblond about 13 years ago Actions #13
- % Done changed from 0 to 90
https://github.com/inliniac/suricata/pull/298 is implementing a workaround based on Victor's analysis. This has been deployed on our test box and seems to work.
PM Updated by Peter Manev about 13 years ago Actions #14
The patch seems to do the job correctly.
VJ Updated by Victor Julien about 13 years ago Actions #15
- Status changed from New to Closed
- % Done changed from 90 to 100
Merged https://github.com/inliniac/suricata/pull/298, thanks guys!
VJ Updated by Victor Julien about 13 years ago Actions #16
- Assignee changed from OISF Dev to Eric Leblond