Project

General

Profile

Actions

Bug #6006

closed

Task #5965: tracking: Improving DPDK capture interface and docs

dpdk: query eth stats only by the first worker

Added by Lukas Sismis 10 months ago. Updated 10 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

When Suricata is under heavy load (and imissed counter is increasing heavily) it leads to buffer-overflow-like behavior when the counter suddenly has very high (unrealistic) values. Example of such behavior can be seen in the counters below when for 4 seconds packets in the imissed counter are increasing by 500k per second and then the counter is suddenly doubled.

After further investigation, I concluded that the function rte_eth_stats_get is not multi-thread safe. (as the behavior was neither reproduced when Suricata was not using this function at all nor when only the first worker used the function).

At the same time, the result of this function is not used anywhere for non-first workers. Therefore, suggested fix is to enable the usage of the stats retrieval to the first worker only.

$ clear && cat /tmp/stats.log | grep -E "Date|capture" 
Date: 4/18/2023 -- 12:09:53 (uptime: 0d, 00h 00m 08s)
Date: 4/18/2023 -- 12:09:55 (uptime: 0d, 00h 00m 10s)
capture.packets                               | Total                     | 4319079841
capture.rx_errors                             | Total                     | 4312173989
capture.dpdk.imissed                          | Total                     | 4312173989
Date: 4/18/2023 -- 12:09:57 (uptime: 0d, 00h 00m 12s)
capture.packets                               | Total                     | 4351338655
capture.rx_errors                             | Total                     | 4334083449
capture.dpdk.imissed                          | Total                     | 4334083449
Date: 4/18/2023 -- 12:09:59 (uptime: 0d, 00h 00m 14s)
capture.packets                               | Total                     | 4351338655
capture.rx_errors                             | Total                     | 4334083449
capture.dpdk.imissed                          | Total                     | 4334083449
Date: 4/18/2023 -- 12:10:01 (uptime: 0d, 00h 00m 16s)
capture.packets                               | Total                     | 8678269587
capture.rx_errors                             | Total                     | 8650362891
capture.dpdk.imissed                          | Total                     | 8650362891
Date: 4/18/2023 -- 12:10:03 (uptime: 0d, 00h 00m 18s)
capture.packets                               | Total                     | 8710930232
capture.rx_errors                             | Total                     | 8672352621
capture.dpdk.imissed                          | Total                     | 8672352621
Date: 4/18/2023 -- 12:10:05 (uptime: 0d, 00h 00m 20s)
capture.packets                               | Total                     | 8710930232
capture.rx_errors                             | Total                     | 8672352621
capture.dpdk.imissed                          | Total                     | 8672352621
Date: 4/18/2023 -- 12:10:07 (uptime: 0d, 00h 00m 22s)
capture.packets                               | Total                     | 17334045626
capture.rx_errors                             | Total                     | 17284806839
capture.dpdk.imissed                          | Total                     | 17284806839
Date: 4/18/2023 -- 12:10:09 (uptime: 0d, 00h 00m 24s)
capture.packets                               | Total                     | 17366979962
capture.rx_errors                             | Total                     | 17307296380
capture.dpdk.imissed                          | Total                     | 17307296380 
Actions

Also available in: Atom PDF