Project

General

Profile

Actions

Optimization #2845

closed

Counters for kernel_packets decreases at times without restart

Added by Eric Urban about 5 years ago. Updated over 3 years ago.

Status:
Closed
Priority:
Normal
Target version:
Effort:
Difficulty:
Label:

Description

We have seen cases in Suricata where the stats.capture.kernel_packets counter decreases while Suricata is running. My understanding is that this is supposed to be a running counter that should not decrease unless Suricata is restarted. This behavior has been observed on 4.0.6 and 4.1.2. I am fairly confident I have also seen this on 3.2.2 as well. This decrease would be more expected if the value reset or rolled over from overflow, but I don't believe that is what is happening here.

Below is one example from the logs I am attaching. I have many other logs I can provide if desired.

$ jq 'select(.event_type == "stats") | select(.timestamp | startswith("2019-02-22T07:55:")) | .timestamp, .stats.capture' eve.json_stats_only_08-snf3-2019022208

...
"2019-02-22T07:55:36.000327-0600" 
{
  "kernel_packets": 17308040184,
  "kernel_packets_delta": 1039779,
  "kernel_drops": 0,
  "kernel_drops_delta": 0,
  "kernel_ifdrops": 0,
  "kernel_ifdrops_delta": 0
}
"2019-02-22T07:55:45.000335-0600" 
{
  "kernel_packets": 13013890235,
  "kernel_packets_delta": -4294149949,
  "kernel_drops": 0,
  "kernel_drops_delta": 0,
  "kernel_ifdrops": 0,
  "kernel_ifdrops_delta": 0
}
"2019-02-22T07:55:54.000320-0600" 
{
  "kernel_packets": 13014866476,
  "kernel_packets_delta": 976241,
  "kernel_drops": 0,
  "kernel_drops_delta": 0,
  "kernel_ifdrops": 0,
  "kernel_ifdrops_delta": 0
}

Corresponding from stats.log:

Date: 2/22/2019 -- 07:55:36 (uptime: 2d, 21h 14m 54s)
------------------------------------------------------------------------------------
Counter                                    | TM Name                   | Value
------------------------------------------------------------------------------------
capture.kernel_packets                     | Total                     | 17308040184
------------------------------------------------------------------------------------

Date: 2/22/2019 -- 07:55:45 (uptime: 2d, 21h 15m 03s)
------------------------------------------------------------------------------------
Counter                                    | TM Name                   | Value
------------------------------------------------------------------------------------
capture.kernel_packets                     | Total                     | 13013890235
------------------------------------------------------------------------------------

Date: 2/22/2019 -- 07:55:54 (uptime: 2d, 21h 15m 12s)
------------------------------------------------------------------------------------
Counter                                    | TM Name                   | Value
------------------------------------------------------------------------------------
capture.kernel_packets                     | Total                     | 13014866476
------------------------------------------------------------------------------------

Here are more examples from other Suricata instances that don't have logs attached, but I am including for reference:

"2019-02-22T15:09:00.000327-0600" 
{
  "kernel_packets": 15681829155,
  "kernel_packets_delta": -4294025171,
  "kernel_drops": 0,
  "kernel_drops_delta": 0,
  "kernel_ifdrops": 0,
  "kernel_ifdrops_delta": 0
}
"2019-02-22T03:18:51.000325-0600" 
{
  "kernel_packets": 15980883154,
  "kernel_packets_delta": -4293598551,
  "kernel_drops": 0,
  "kernel_drops_delta": 0,
  "kernel_ifdrops": 0,
  "kernel_ifdrops_delta": 0
}
"2019-02-19T10:22:00.000363-0600" 
{
  "kernel_packets": 17749102321,
  "kernel_packets_delta": -4294216445,
  "kernel_drops": 2227794327,
  "kernel_drops_delta": 0,
  "kernel_ifdrops": 0,
  "kernel_ifdrops_delta": 0
}
"2019-02-19T10:17:40.000327-0600" 
{
  "kernel_packets": 16791755239,
  "kernel_packets_delta": -4294006615,
  "kernel_drops": 1280457873,
  "kernel_drops_delta": 0,
  "kernel_ifdrops": 0,
  "kernel_ifdrops_delta": 0
}
"2019-02-19T09:30:35.000346-0600" 
{
  "kernel_packets": 17342905685,
  "kernel_packets_delta": -4294369072,
  "kernel_drops": 580833306,
  "kernel_drops_delta": 0,
  "kernel_ifdrops": 0,
  "kernel_ifdrops_delta": 0
}
"2019-02-19T09:25:05.000338-0600" 
{
  "kernel_packets": 23570036423,
  "kernel_packets_delta": -4293688281,
  "kernel_drops": 775213362,
  "kernel_drops_delta": 0,
  "kernel_ifdrops": 0,
  "kernel_ifdrops_delta": 0
}
"2019-02-19T08:51:53.000331-0600" 
{
  "kernel_packets": 12005768232,
  "kernel_packets_delta": -4294159125,
  "kernel_drops": 4547641950,
  "kernel_drops_delta": 0,
  "kernel_ifdrops": 0,
  "kernel_ifdrops_delta": 0
}
"2019-02-19T08:51:03.000358-0600" 
{
  "kernel_packets": 22256188092,
  "kernel_packets_delta": -4294023378,
  "kernel_drops": 722622375,
  "kernel_drops_delta": 0,
  "kernel_ifdrops": 0,
  "kernel_ifdrops_delta": 0
}

I do not see any messages in the suricata.log file during this time.

Is this behavior expected and if not what additional troubleshooting would you like us to perform to assist with this issue?


Files

stats_08-snf3.log-2019022208.gz (124 KB) stats_08-snf3.log-2019022208.gz Stats log Eric Urban, 02/22/2019 09:11 PM
eve.json_stats_only_08-snf3-2019022208.gz (203 KB) eve.json_stats_only_08-snf3-2019022208.gz Eve log (filtered for only stats events) Eric Urban, 02/22/2019 09:12 PM
stats.log-2019030413.gz (2.68 MB) stats.log-2019030413.gz Eric Urban, 03/05/2019 06:51 PM
Actions

Also available in: Atom PDF