Feature #7036
closedFeature #6805: cpu-affinity: enhance CPU affinity logic with per-interface NUMA preferences
DPDK NUMA setup: choose correct CPUs from worker-cpu-set
Description
My machine has even numbered cores on NUMA0 and odd numbered cores on NUMA1. I have defined two vdev interfaces, one on each socket, and defined worker-cpu-set as:cpu: [ 2,4,6,8,10,12,14,3,5,7,9,11,13,15 ]
The desired behaviour is that first 7 threads (2,4,6,8,10,12,14) use NIC on NUMA0 and the remaining 7 (3,5,7,9,11,13,15) use NIC on NUMA1.
Actual behaviour is that the cpu list is probably sorted internally and read as [ 2,3,4,5,6,7,8,9,10,11,12,13,14,15 ]. As a result, I receive a warning stating - dpdk: net_bonding0: NIC is on NUMA 0, 3 threads on different NUMA node(s) , which indicates that threads 2-8 are using NIC on NUMA0 and 9-15 are using NIC on NUMA1.
AK Updated by Adhirath Kabra almost 2 years ago
- Subject changed from NUMA: choose correct CPUs from worker-cpu-set to DPDK NUMA setup: choose correct CPUs from worker-cpu-set
- Assignee changed from OISF Dev to Lukas Sismis
LS Updated by Lukas Sismis almost 2 years ago
- Tracker changed from Bug to Feature
- Status changed from New to Assigned
- Target version changed from TBD to 8.0.0-beta1
- Parent task set to #6805
- Affected Versions deleted (
7.0.5)
LS Updated by Lukas Sismis almost 2 years ago
- Status changed from Assigned to In Progress
VJ Updated by Victor Julien almost 2 years ago
- Related to Task #3318: Research: NUMA awareness added
VJ Updated by Victor Julien almost 2 years ago
- Related to Bug #7137: "invalid cpu range" when trying to use CPU affinity added
LS Updated by Lukas Sismis over 1 year ago
Here is the suggestion for the feature.
// threading.cpu-assignment:
// - legacy - assign as usual
// - auto - use hwloc to determine NUMA locality of the NIC and try to assign a core from this NUMA node.
// If it fails then use the other NUMA node.
// Using this approach e.g. on bonded devices/aliased and any other will not work
// Warn/Notify a user when device's NUMA node cannot be determined.
// Mention in the docs that NUMA locatity supports PCIe addresses and Kernel interfaces
// - manual - in workers CPU set either:
// - Specify in one line ([ "eth0@1,2,3,4,7-9", "eth1@10,11" ])
// - Specify threading in a list:
// - worker-cpu-set:
// - interface: eth0
// cpu: [ 1,2,3,4 ]
// mode: "exclusive"
// prio:
// high: [ 3 ]
// default: "medium"
LS Updated by Lukas Sismis over 1 year ago ยท Edited
Current draft containing something like auto mode (not fully implemented yet):
https://github.com/OISF/suricata/compare/master...lukashino:suricata:feat/6805-numa-cpu-locality-v1
(Manual support not included)
AK Updated by Adhirath Kabra over 1 year ago
Lukas Sismis wrote in #note-6:
Here is the suggestion for the feature.
[...]
Nice suggestion, especially the flexibility in resource allocation that manual mode would offer. For example as an IDS, monitored links can be assigned CPUs proportionate to the amount of average traffic they have.
VJ Updated by Victor Julien over 1 year ago
Lukas Sismis wrote in #note-7:
Current draft containing something like auto mode (not fully implemented yet):
https://github.com/OISF/suricata/compare/master...lukashino:suricata:feat/6805-numa-cpu-locality-v1(Manual support not included)
Please do this as a draft PR so we can discuss implementation things there.
LS Updated by Lukas Sismis over 1 year ago
Just created:
https://github.com/OISF/suricata/pull/11521
LS Updated by Lukas Sismis over 1 year ago
New update is here - https://github.com/OISF/suricata/pull/11706
It has been polished and now I have added a manual setting as well.
VJ Updated by Victor Julien over 1 year ago
- Related to Task #7336: Suricon 2024 brainstorm added
LS Updated by Lukas Sismis about 1 year ago
- Status changed from In Progress to In Review
VJ Updated by Victor Julien about 1 year ago
- Target version changed from 8.0.0-beta1 to 8.0.0-rc1
LS Updated by Lukas Sismis 10 months ago
- Status changed from In Review to Closed