Project

General

Profile

Actions

Bug #2718

open

pkts/drops counters discrepancy

Added by Peter Manev over 5 years ago. Updated over 4 years ago.

Status:
New
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

Running git with AFPv3 (info below at the end) and seeing discrepancy in total and dropped packet counters on the same Suricata run between suricata.log/stdout and stats.log/eve stats.json
Seeing this on 2 different high speed Suricata deployments.


------------------------------------------------------------------------------------
Date: 11/29/2018 -- 22:53:39 (uptime: 0d, 20h 58m 32s)
------------------------------------------------------------------------------------
Counter                                    | TM Name                   | Value
------------------------------------------------------------------------------------
capture.kernel_packets                     | Total                     | 34526257089
capture.kernel_drops                       | Total                     | 364864207
decoder.pkts                               | Total                     | 34161381126
decoder.bytes                              | Total                     | 27521454249680
decoder.invalid                            | Total                     | 125528
decoder.ipv4                               | Total                     | 33874924190
decoder.ipv6                               | Total                     | 299143889
decoder.ethernet                           | Total                     | 34161381126

[1841] 29/11/2018 -- 22:53:39 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#15-eth3) Kernel: Packets 1748200153, dropped 23108264
[1841] 29/11/2018 -- 22:53:39 - (log-filestore.c:568) <Info> (LogFilestoreLogExitPrintStats) -- (W#15-eth3) Files extracted 0
[1842] 29/11/2018 -- 22:53:39 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#16-eth3) Kernel: Packets 13, dropped 0
[1842] 29/11/2018 -- 22:53:39 - (log-filestore.c:568) <Info> (LogFilestoreLogExitPrintStats) -- (W#16-eth3) Files extracted 0
[1843] 29/11/2018 -- 22:53:39 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#01-eth0) Kernel: Packets 1235440, dropped 119484
[1843] 29/11/2018 -- 22:53:39 - (log-filestore.c:568) <Info> (LogFilestoreLogExitPrintStats) -- (W#01-eth0) Files extracted 0
[1701] 29/11/2018 -- 22:53:39 - (counters.c:830) <Info> (StatsLogSummary) -- Alerts: 3052
[1701] 29/11/2018 -- 22:53:40 - (ippair.c:290) <Perf> (IPPairPrintStats) -- ippair memory usage: 414144 bytes, maximum: 16777216
[1701] 29/11/2018 -- 22:53:42 - (host.c:294) <Perf> (HostPrintStats) -- host memory usage: 1622144 bytes, maximum: 16777216
[1701] 29/11/2018 -- 22:53:42 - (detect-engine-build.c:1733) <Info> (SigAddressCleanupStage1) -- cleaning up signature grouping structure... complete
[1701] 29/11/2018 -- 22:53:42 - (util-device.c:329) <Notice> (LiveDeviceListClean) -- Stats for 'eth2':  pkts: 8251080725, drop: 42475597 (0.51%), invalid chksum: 0
[1701] 29/11/2018 -- 22:53:42 - (util-device.c:329) <Notice> (LiveDeviceListClean) -- Stats for 'eth3':  pkts: 26275081393, drop: 322269126 (1.23%), invalid chksum: 0
[1701] 29/11/2018 -- 22:53:42 - (util-device.c:329) <Notice> (LiveDeviceListClean) -- Stats for 'eth0':  pkts: 1235440, drop: 119484 (9.67%), invalid chksum: 0
[1701] 29/11/2018 -- 22:53:43 - (util-mpm-hs.c:1070) <Perf> (MpmHSGlobalCleanup) -- Cleaning up Hyperscan global scratch
[1701] 29/11/2018 -- 22:53:43 - (util-mpm-hs.c:1078) <Perf> (MpmHSGlobalCleanup) -- Clearing Hyperscan database cache
root@suricata:~#

Above -
Total counter(s, combined) has a mismatch
Drop counter(s , combined) is good - exactly the same


[74190] 29/11/2018 -- 23:06:27 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#33-enp59s0) Kernel: Packets 1063135836, dropped 1969930
[74191] 29/11/2018 -- 23:06:27 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#34-enp59s0) Kernel: Packets 1069756460, dropped 4021948
[74192] 29/11/2018 -- 23:06:27 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#35-enp59s0) Kernel: Packets 1128621774, dropped 1065953
[74193] 29/11/2018 -- 23:06:27 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#36-enp59s0) Kernel: Packets 1143644762, dropped 1572917
[74194] 29/11/2018 -- 23:06:27 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#37-enp59s0) Kernel: Packets 1029388098, dropped 110874
[74195] 29/11/2018 -- 23:06:27 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#38-enp59s0) Kernel: Packets 1049514375, dropped 237793
[74196] 29/11/2018 -- 23:06:28 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#39-enp59s0) Kernel: Packets 1109232251, dropped 2193864
[74197] 29/11/2018 -- 23:06:28 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#40-enp59s0) Kernel: Packets 1125246426, dropped 1889525
[73968] 29/11/2018 -- 23:06:28 - (counters.c:830) <Info> (StatsLogSummary) -- Alerts: 16159124
[73968] 29/11/2018 -- 23:06:37 - (ippair.c:290) <Perf> (IPPairPrintStats) -- ippair memory usage: 414144 bytes, maximum: 16777216
[73968] 29/11/2018 -- 23:06:45 - (host.c:294) <Perf> (HostPrintStats) -- host memory usage: 33554400 bytes, maximum: 33554432
[73968] 29/11/2018 -- 23:06:48 - (detect-engine-build.c:1733) <Info> (SigAddressCleanupStage1) -- cleaning up signature grouping structure... complete
[73968] 29/11/2018 -- 23:06:48 - (util-device.c:329) <Notice> (LiveDeviceListClean) -- Stats for 'enp59s0':  pkts: 44861666289, drop: 106594312 (0.24%), invalid chksum: 0
[73968] 29/11/2018 -- 23:06:48 - (util-mpm-hs.c:1070) <Perf> (MpmHSGlobalCleanup) -- Cleaning up Hyperscan global scratch
[73968] 29/11/2018 -- 23:06:48 - (util-mpm-hs.c:1078) <Perf> (MpmHSGlobalCleanup) -- Clearing Hyperscan database cache

------------------------------------------------------------------------------------
Date: 11/29/2018 -- 23:06:25 (uptime: 0d, 06h 24m 52s)
------------------------------------------------------------------------------------
Counter                                    | TM Name                   | Value
------------------------------------------------------------------------------------
capture.kernel_packets                     | Total                     | 44852339658
capture.kernel_drops                       | Total                     | 103678933
decoder.pkts                               | Total                     | 44748644246
decoder.bytes                              | Total                     | 42596317608384
decoder.invalid                            | Total                     | 23390
decoder.ipv4                               | Total                     | 39668212206
decoder.ipv6                               | Total                     | 6252968947
decoder.ethernet                           | Total                     | 44748644246
decoder.tcp                                | Total                     | 34949571497

Here both counters differ (in millions) - total and dropped.

Also even in a short run as well - where it seems there is even bigger diff.

[41642] 29/11/2018 -- 23:38:46 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#38-enp59s0) Kernel: Packets 39079572, dropped 229553
[41643] 29/11/2018 -- 23:38:46 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#39-enp59s0) Kernel: Packets 41426017, dropped 306604
[41644] 29/11/2018 -- 23:38:47 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#40-enp59s0) Kernel: Packets 35466864, dropped 215202
[41420] 29/11/2018 -- 23:38:47 - (counters.c:830) <Info> (StatsLogSummary) -- Alerts: 440925
[41420] 29/11/2018 -- 23:38:51 - (ippair.c:290) <Perf> (IPPairPrintStats) -- ippair memory usage: 414144 bytes, maximum: 16777216
[41420] 29/11/2018 -- 23:38:53 - (host.c:294) <Perf> (HostPrintStats) -- host memory usage: 1746992 bytes, maximum: 33554432
[41420] 29/11/2018 -- 23:38:55 - (detect-engine-build.c:1733) <Info> (SigAddressCleanupStage1) -- cleaning up signature grouping structure... complete
[41420] 29/11/2018 -- 23:38:55 - (util-device.c:329) <Notice> (LiveDeviceListClean) -- Stats for 'enp59s0':  pkts: 1563131426, drop: 10799950 (0.69%), invalid chksum: 0
[41420] 29/11/2018 -- 23:38:55 - (util-mpm-hs.c:1070) <Perf> (MpmHSGlobalCleanup) -- Cleaning up Hyperscan global scratch
[41420] 29/11/2018 -- 23:38:55 - (util-mpm-hs.c:1078) <Perf> (MpmHSGlobalCleanup) -- Clearing Hyperscan database cache

------------------------------------------------------------------------------------
Date: 11/29/2018 -- 23:38:44 (uptime: 0d, 00h 18m 29s)
------------------------------------------------------------------------------------
Counter                                    | TM Name                   | Value
------------------------------------------------------------------------------------
capture.kernel_packets                     | Total                     | 1548885068
capture.kernel_drops                       | Total                     | 1766298
decoder.pkts                               | Total                     | 1547105082
decoder.bytes                              | Total                     | 1535747270554
decoder.invalid                            | Total                     | 1584
decoder.ipv4                               | Total                     | 1424313847
decoder.ipv6                               | Total                     | 181882866
decoder.ethernet                           | Total                     | 1547105082

Some more info

This is Suricata version 4.1.0-dev (rev 683be948)
Features: PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LUA HAVE_LUAJIT HAVE_LIBJANSSON TLS MAGIC RUST 
SIMD support: SSE_4_2 SSE_4_1 SSE_3 
Atomic intrisics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 4.2.1 Compatible Clang 6.0.0 (tags/RELEASE_600/final), C version 199901
compiled with _FORTIFY_SOURCE=0
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.28, linked against LibHTP v0.5.28

Suricata Configuration:
  AF_PACKET support:                       yes
  eBPF support:                            yes
  XDP support:                             yes
  PF_RING support:                         no
  NFQueue support:                         no
  NFLOG support:                           no
  IPFW support:                            no
  Netmap support:                          no
  DAG enabled:                             no
  Napatech enabled:                        no
  WinDivert enabled:                       no

  Unix socket enabled:                     yes
  Detection enabled:                       yes

  Libmagic support:                        yes
  libnss support:                          yes
  libnspr support:                         yes
  libjansson support:                      yes
  liblzma support:                         no
  hiredis support:                         no
  hiredis async with libevent:             no
  Prelude support:                         no
  PCRE jit:                                yes
  LUA support:                             yes, through luajit
  libluajit:                               yes
  libgeoip:                                yes
  Non-bundled htp:                         no
  Old barnyard2 support:                   no
  Hyperscan support:                       yes
  Libnet support:                          yes
  liblz4 support:                          yes

  Rust support:                            yes (default)
  Rust strict mode:                        yes
  Rust debug mode:                         no
  Rust compiler:                           rustc 1.24.1
  Rust cargo:                              cargo 0.25.0

  Suricatasc install:                      yes

  Profiling enabled:                       no
  Profiling locks enabled:                 no

Development settings:
  Coccinelle / spatch:                     yes
  Unit tests enabled:                      no
  Debug output enabled:                    no
  Debug validation enabled:                no

Generic build parameters:
  Installation prefix:                     /usr
  Configuration directory:                 /etc/suricata/
  Log directory:                           /var/log/suricata/

  --prefix                                 /usr
  --sysconfdir                             /etc
  --localstatedir                          /var

  Host:                                    x86_64-pc-linux-gnu
  Compiler:                                clang-6.0 (exec name) / clang (real)
  GCC Protect enabled:                     no
  GCC march native enabled:                yes
  GCC Profile enabled:                     no
  Position Independent Executable enabled: no
  CFLAGS                                   -g -O2 -march=native -I${srcdir}/../rust/gen/c-headers
  PCAP_CFLAGS                               -I/usr/include
  SECCFLAGS                                

Actions #1

Updated by Victor Julien over 5 years ago

One thing that is less than ideal is that afpacket has 2 places it updates counters: the Stats API and the Livedev. Both are updated from the same source, but it remains a duplicate effort.

Actions #2

Updated by Andreas Herz almost 5 years ago

  • Assignee set to OISF Dev
  • Target version set to TBD
Actions #3

Updated by Andreas Herz over 4 years ago

What would be a proper solution for this?

Actions

Also available in: Atom PDF