Project

General

Profile

Actions

Bug #3049

closed

thread hangs in pfring mode

Added by Arthur Kokhanenko about 5 years ago. Updated almost 5 years ago.

Status:
Closed
Priority:
Normal
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

Under high load capturing thread sometimes hangs.
It doesn't consume CPU after that
  1. ps -T -p 18131
    PID SPID TTY TIME CMD
    18131 18131 ? 00:18:18 Suricata-Main
    18131 18132 ? 3-15:32:14 RX#01-ens1
    18131 18133 ? 3-11:31:03 RX#02-ens1
    18131 18134 ? 2-17:07:14 RX#03-ens1
    18131 18135 ? 3-16:46:10 RX#04-ens1
    and at pfring stats all packets are dropped
  2. cat /proc/net/pf_ring/18134-ens1.8
    Tot Packets : 122565158041
    Tot Pkt Lost : 22083715992
    other threads fine:
  3. cat /proc/net/pf_ring/18133-ens1.7
    Tot Packets : 130281268130
    Tot Pkt Lost : 213073812

Configuration:
- interface: default
threads: 4
- interface: ens1
cluster-id: 10
cluster-type: cluster_flow

After connecting gdb to suricata, bt from hanged thread:
(gdb) thr 55
[Switching to thread 55 (Thread 0x7f49154dc700 (LWP 18134))]
#0 0x00007f491d54f965 in pthread_cond_wait@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) bt
#0 0x00007f491d54f965 in pthread_cond_wait
@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00005652590f965a in PacketPoolWait () at tmqh-packetpool.c:155
#2 0x00005652590e3c1f in ReceivePfringLoop (tv=0x56525d739f70, data=0x7f48f09bc8c0, slot=<optimized out>) at source-pfring.c:347
#3 0x00005652590fe0c7 in TmThreadsSlotPktAcqLoop (td=0x56525d739f70) at tm-threads.c:334
#4 0x00007f491d54bdd5 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f491ce5aead in clone () from /lib64/libc.so.6

Probably issue here:
void PacketPoolWait(void) {
PktPool *my_pool = GetThreadPacketPool();

if (PacketPoolIsEmpty(my_pool)) {
SCMutexLock(&my_pool->return_stack.mutex);

// <== you have to check again PacketPoolIsEmpty here, otherwise nobody to send Signal here

SC_ATOMIC_ADD(my_pool->return_stack.sync_now, 1);
SCCondWait(&my_pool->return_stack.cond, &my_pool->return_stack.mutex);
SCMutexUnlock(&my_pool->return_stack.mutex);
}
while(PacketPoolIsEmpty(my_pool))
cc_barrier();
}
Actions #1

Updated by Andreas Herz about 5 years ago

  • Assignee set to Community Ticket
  • Priority changed from High to Normal
  • Target version set to TBD
  • Effort deleted (high)

Can you try to reproduce it with a current 4.1.x version?

Actions #2

Updated by Peter Manev about 5 years ago

Also - i could not see the pfring version you are using.

Actions #3

Updated by Arthur Kokhanenko about 5 years ago

we will check with suricata 4.1.4
pfring-7.5.0-2417.x86_64

Actions #4

Updated by Andreas Herz almost 5 years ago

Could you reproduce it at the test with the new versions?

Actions #5

Updated by Arthur Kokhanenko almost 5 years ago

Looks like unable to reproduce with 4.1.4

Actions #6

Updated by Andreas Herz almost 5 years ago

  • Status changed from New to Closed

Thanks for the feedback!

Actions

Also available in: Atom PDF