Bug #1777: asymmetrical af_packet hash function in recent kernel - Suricata - Open Information Security Foundation

Actions

Copy link

Bug #1777

closed

asymmetrical af_packet hash function in recent kernel

Added by Eric Leblond almost 8 years ago. Updated over 7 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Target version:

Affected Versions:

Effort:

Difficulty:

Label:

Description

The in kernel hash function is not symmetrical anymore since the 4.2 kernel. This introduces out of order treatment of packets. A possible solution is to used the new cluster_ebpf load balancing mode to introduce our own hashing functions.

Actions

Copy link

Updated by Victor Julien almost 8 years ago

Status changed from New to Assigned
Priority changed from Normal to High
Target version set to 70

Actions

Copy link

Updated by Derek Ditch almost 8 years ago

I filed a bug with the kernel bugzilla. I don't have solid reproducible input at the moment, but I don't see any reason why the new hashing to include vlan and flow-labels can't be symmetric as well.

https://bugzilla.kernel.org/show_bug.cgi?id=120441

Actions

Copy link

Updated by Victor Julien almost 8 years ago

If I'm not mistaken, we can rely on the NIC to do the balancing:

$ ethtool -n ens2f0 rx-flow-hash tcp4
TCP over IPV4 flows use these fields for computing Hash flow key:
IP SA
IP DA
L4 bytes 0 & 1 [TCP/UDP src port]
L4 bytes 2 & 3 [TCP/UDP dst port]

$ ethtool -n ens2f0 rx-flow-hash tcp6
TCP over IPV6 flows use these fields for computing Hash flow key:
IP SA
IP DA
L4 bytes 0 & 1 [TCP/UDP src port]
L4 bytes 2 & 3 [TCP/UDP dst port]

$ ethtool -n ens2f0 rx-flow-hash udp4
UDP over IPV4 flows use these fields for computing Hash flow key:
IP SA
IP DA
L4 bytes 0 & 1 [TCP/UDP src port]
L4 bytes 2 & 3 [TCP/UDP dst port]

$ ethtool -n ens2f0 rx-flow-hash udp6
UDP over IPV6 flows use these fields for computing Hash flow key:
IP SA
IP DA
L4 bytes 0 & 1 [TCP/UDP src port]
L4 bytes 2 & 3 [TCP/UDP dst port]

Then instead of 'cluster_flow' in the yaml set 'cluster_qm'.

Actions

Copy link

Updated by Victor Julien almost 8 years ago

It seems my thinking about RSS is wrong. As far I understand now, the RSS hashing is not symmetric either. It makes sense if you consider that the promisc capture case is a bit of an anomaly: normally the RX of a NIC would not see both sides of a flow. See also http://www.ntop.org/pf_ring/going-beyond-rss-receive-side-scaling/

Actions

Copy link

Updated by Eric Leblond almost 8 years ago

For xl710, section 7.1.9.3 of xl710-10-40-controller-datasheet.pdf announced it is possible to get a symetrical hash. From conversation I had at netdev 1.1 with Intel people it was possible to have it.

Actions

Copy link

Updated by Victor Julien almost 8 years ago

David Miller has provided a fix that we're currently testing. It should go into stable kernels as well once it's ready, as the change seems not intrusive. That should fix the issue.

However, it seems that generally using multiple RSS queues is harmful. Most NIC's will not do symmetric hashing on the NIC leading to both sides of the conversation possibly being sent to different queues. Timing issue can lead to packet ordering problems. The solution here is to set the number of queues to 1 or force the NIC to use symmetric hashing if possible.

Actions

Copy link