Project

General

Profile

Actions
Copy link

Feature #3011

closed
CN EL

Add new 'cluster_peer' runmode to allow for load balancing by IP header (src<->dst) only

Feature #3011: Add new 'cluster_peer' runmode to allow for load balancing by IP header (src<->dst) only

Added by Cooper Nelson almost 7 years ago. Updated over 6 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Eric Leblond
Target version:
5.0rc1
Effort:
Difficulty:
Label:

Description

I'm investigating an issue on a production deployment that is seeing a large number of 'tcp.pkt_on_wrong_thread' in stats.log.

My current theory is that this is due to fragmented TCP packets not being properly hashed by the kernels RSS implementation and sent to different cores/threads.

One idea I had to address this was to a new cluster runmode that simply load-balanced based on the IP header only, so even if the packets were on the 'wrong' RSS queue, they would be directed to the same worker thread. However it's still possible/likely that the fragments will end up in the wrong order on the worker thread, which may cause other issues.

The 'right' way to fix this is to force the hashing on the NIC itself, however I'm not sure if that is possible in all cases.

Related issues 2 (1 open — 1 closed)

Related to Suricata - Optimization #2725: stream/packet on wrong threadFeedbackOISF DevActions
Related to Suricata - Bug #1778: af_packet: IPS and defragClosedEric LeblondActions

AH Updated by Andreas Herz almost 7 years ago Actions
Copy link
 #1

  • Assignee set to Community Ticket
  • Target version set to TBD

VJ Updated by Victor Julien almost 7 years ago Actions
Copy link
 #2

VJ Updated by Victor Julien almost 7 years ago Actions
Copy link
 #3

It doesn't look like AF_PACKET has the support for this. See https://github.com/torvalds/linux/blob/master/net/packet/af_packet.c#L1419 for the built-in options. I think the way to do this would be through eBPF.

EL Updated by Eric Leblond almost 7 years ago Actions
Copy link
 #4

By using cluster_ebpf and the provided lb.pdf file, you will have IP pair load balancing done by the kernel. The documentation on usage is here: https://suricata.readthedocs.io/en/suricata-4.1.4/capture-hardware/ebpf-xdp.html#setup-ebpf-load-balancing

VJ Updated by Victor Julien over 6 years ago Actions
Copy link
 #5

  • Status changed from New to Closed
  • Assignee changed from Community Ticket to Eric Leblond
  • Target version changed from TBD to 5.0rc1

Considering this done as it can't be done using regular AF_PACKET, so the eBPF way is the only way we can support.

VJ Updated by Victor Julien over 6 years ago Actions
Copy link
 #6

  • Related to Bug #1778: af_packet: IPS and defrag added
Actions
Copy link

Also available in: PDF Atom