Project

General

Profile

Actions

Optimization #2725

open

stream/packet on wrong thread

Added by Peter Manev over 5 years ago. Updated 2 days ago.

Status:
Feedback
Priority:
Normal
Assignee:
Target version:
Effort:
Difficulty:
Label:

Description

Looking for feedback.

While investigating various research points with af-packet on live traffic and latest gitmaster (ex eedf08be/4.1) I noticed that I have never seen those to be 0 in any occasion

stream.wrong_thread                        | Total                     | 2982446
tcp.pkt_on_wrong_thread                    | Total                     | 156187846

Those statistics can be made available via - ( https://github.com/OISF/suricata/blob/master/suricata.yaml.in#L66 )

# global stats configuration
stats:
  enabled: yes
  # The interval field (in seconds) controls at what interval
  # the loggers are invoked.
  interval: 8
  # Add decode events as stats.
  decoder-events: true
  # Add stream events as stats.
  stream-events: true

I have tried different NICs/drivers(tested ixgbe/i40e), af-packet v3/v2, cluster_flow/cluster_cpu/cluster_qm, vlan tracking enabled or not, on different live traffic machines, different kernels (4.18/4.19) -
capture.kernel_drops and stream.wrong_thread are never 0 and always increasing.(it is more like 10-15% of the total in my test cases)

Looking for any feedback in terms of - if you are experiencing the same issue or not and what is your setup (if you would like to share).


Files

statslog.tar.gz (802 KB) statslog.tar.gz Peter Manev, 12/05/2018 07:37 AM
wrong_threads.png (43.4 KB) wrong_threads.png Andreas Herz, 05/03/2019 08:53 AM
WRONG_THREAD.ods (13.4 KB) WRONG_THREAD.ods Sean Cloherty, 05/23/2019 07:59 PM
stats.log (10.7 KB) stats.log Sean Cloherty, 05/31/2019 02:19 PM
issue-2725.tar.xz (99.6 KB) issue-2725.tar.xz Peter Manev, 06/11/2019 08:18 PM
Screenshot from 2019-06-23 11-04-54.png (228 KB) Screenshot from 2019-06-23 11-04-54.png Peter Manev, 06/24/2019 06:14 AM
excerpt.pcap (37.2 KB) excerpt.pcap Produces tcp.pkt_on_wrong_thread Gatewatcher Dev Team, 07/02/2019 03:56 PM
http_png.pcap (19.1 KB) http_png.pcap HTTP Download OK Gatewatcher Dev Team, 07/02/2019 03:57 PM
rps_http_png.pcap (19.3 KB) rps_http_png.pcap Produces tcp.pkt_on_wrong_thread with vanilla RPS implementation Gatewatcher Dev Team, 07/05/2019 01:03 PM

Related issues 8 (3 open5 closed)

Related to Suricata - Support #2900: alert 'SURICATA STREAM pkt seen on wrong thread' when run mode set to workersClosedOISF DevActions
Related to Suricata - Feature #3011: Add new 'cluster_peer' runmode to allow for load balancing by IP header (src<->dst) onlyClosedEric LeblondActions
Related to Suricata - Bug #3158: 'wrong thread' tracking inaccurate for bridging IPS modesClosedVictor JulienActions
Related to Suricata - Feature #3319: on 'wrong thread' reinject packets to correct threadNewOISF DevActions
Related to Suricata - Task #5488: Suricon 2022 brainstormAssignedVictor JulienActions
Related to Suricata - Bug #5270: Flow hash table collision and flow state corruption between different capture interfacesClosedPhilippe AntoineActions
Related to Suricata - Feature #5673: capture: option to decapsulate everything firstNewOISF DevActions
Related to Suricata - Bug #6394: Sudden increase in capture.kernel_drops and tcp.pkt_on_wrong_thread after upgrading to 6.0.14ClosedPhilippe AntoineActions
Actions

Also available in: Atom PDF