Project

General

Profile

Actions

Bug #3049

closed

thread hangs in pfring mode

Added by Arthur Kokhanenko almost 5 years ago. Updated over 4 years ago.

Status:
Closed
Priority:
Normal
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

Under high load capturing thread sometimes hangs.
It doesn't consume CPU after that
  1. ps -T -p 18131
    PID SPID TTY TIME CMD
    18131 18131 ? 00:18:18 Suricata-Main
    18131 18132 ? 3-15:32:14 RX#01-ens1
    18131 18133 ? 3-11:31:03 RX#02-ens1
    18131 18134 ? 2-17:07:14 RX#03-ens1
    18131 18135 ? 3-16:46:10 RX#04-ens1
    and at pfring stats all packets are dropped
  2. cat /proc/net/pf_ring/18134-ens1.8
    Tot Packets : 122565158041
    Tot Pkt Lost : 22083715992
    other threads fine:
  3. cat /proc/net/pf_ring/18133-ens1.7
    Tot Packets : 130281268130
    Tot Pkt Lost : 213073812

Configuration:
- interface: default
threads: 4
- interface: ens1
cluster-id: 10
cluster-type: cluster_flow

After connecting gdb to suricata, bt from hanged thread:
(gdb) thr 55
[Switching to thread 55 (Thread 0x7f49154dc700 (LWP 18134))]
#0 0x00007f491d54f965 in pthread_cond_wait@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) bt
#0 0x00007f491d54f965 in pthread_cond_wait
@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00005652590f965a in PacketPoolWait () at tmqh-packetpool.c:155
#2 0x00005652590e3c1f in ReceivePfringLoop (tv=0x56525d739f70, data=0x7f48f09bc8c0, slot=<optimized out>) at source-pfring.c:347
#3 0x00005652590fe0c7 in TmThreadsSlotPktAcqLoop (td=0x56525d739f70) at tm-threads.c:334
#4 0x00007f491d54bdd5 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f491ce5aead in clone () from /lib64/libc.so.6

Probably issue here:
void PacketPoolWait(void) {
PktPool *my_pool = GetThreadPacketPool();

if (PacketPoolIsEmpty(my_pool)) {
SCMutexLock(&my_pool->return_stack.mutex);

// <== you have to check again PacketPoolIsEmpty here, otherwise nobody to send Signal here

SC_ATOMIC_ADD(my_pool->return_stack.sync_now, 1);
SCCondWait(&my_pool->return_stack.cond, &my_pool->return_stack.mutex);
SCMutexUnlock(&my_pool->return_stack.mutex);
}
while(PacketPoolIsEmpty(my_pool))
cc_barrier();
}
Actions

Also available in: Atom PDF