Optimization #2845
closedCounters for kernel_packets decreases at times without restart
Description
We have seen cases in Suricata where the stats.capture.kernel_packets counter decreases while Suricata is running. My understanding is that this is supposed to be a running counter that should not decrease unless Suricata is restarted. This behavior has been observed on 4.0.6 and 4.1.2. I am fairly confident I have also seen this on 3.2.2 as well. This decrease would be more expected if the value reset or rolled over from overflow, but I don't believe that is what is happening here.
Below is one example from the logs I am attaching. I have many other logs I can provide if desired.
$ jq 'select(.event_type == "stats") | select(.timestamp | startswith("2019-02-22T07:55:")) | .timestamp, .stats.capture' eve.json_stats_only_08-snf3-2019022208
...
"2019-02-22T07:55:36.000327-0600"
{
"kernel_packets": 17308040184,
"kernel_packets_delta": 1039779,
"kernel_drops": 0,
"kernel_drops_delta": 0,
"kernel_ifdrops": 0,
"kernel_ifdrops_delta": 0
}
"2019-02-22T07:55:45.000335-0600"
{
"kernel_packets": 13013890235,
"kernel_packets_delta": -4294149949,
"kernel_drops": 0,
"kernel_drops_delta": 0,
"kernel_ifdrops": 0,
"kernel_ifdrops_delta": 0
}
"2019-02-22T07:55:54.000320-0600"
{
"kernel_packets": 13014866476,
"kernel_packets_delta": 976241,
"kernel_drops": 0,
"kernel_drops_delta": 0,
"kernel_ifdrops": 0,
"kernel_ifdrops_delta": 0
}
Corresponding from stats.log:
Date: 2/22/2019 -- 07:55:36 (uptime: 2d, 21h 14m 54s) ------------------------------------------------------------------------------------ Counter | TM Name | Value ------------------------------------------------------------------------------------ capture.kernel_packets | Total | 17308040184 ------------------------------------------------------------------------------------ Date: 2/22/2019 -- 07:55:45 (uptime: 2d, 21h 15m 03s) ------------------------------------------------------------------------------------ Counter | TM Name | Value ------------------------------------------------------------------------------------ capture.kernel_packets | Total | 13013890235 ------------------------------------------------------------------------------------ Date: 2/22/2019 -- 07:55:54 (uptime: 2d, 21h 15m 12s) ------------------------------------------------------------------------------------ Counter | TM Name | Value ------------------------------------------------------------------------------------ capture.kernel_packets | Total | 13014866476 ------------------------------------------------------------------------------------
Here are more examples from other Suricata instances that don't have logs attached, but I am including for reference:
"2019-02-22T15:09:00.000327-0600"
{
"kernel_packets": 15681829155,
"kernel_packets_delta": -4294025171,
"kernel_drops": 0,
"kernel_drops_delta": 0,
"kernel_ifdrops": 0,
"kernel_ifdrops_delta": 0
}
"2019-02-22T03:18:51.000325-0600"
{
"kernel_packets": 15980883154,
"kernel_packets_delta": -4293598551,
"kernel_drops": 0,
"kernel_drops_delta": 0,
"kernel_ifdrops": 0,
"kernel_ifdrops_delta": 0
}
"2019-02-19T10:22:00.000363-0600"
{
"kernel_packets": 17749102321,
"kernel_packets_delta": -4294216445,
"kernel_drops": 2227794327,
"kernel_drops_delta": 0,
"kernel_ifdrops": 0,
"kernel_ifdrops_delta": 0
}
"2019-02-19T10:17:40.000327-0600"
{
"kernel_packets": 16791755239,
"kernel_packets_delta": -4294006615,
"kernel_drops": 1280457873,
"kernel_drops_delta": 0,
"kernel_ifdrops": 0,
"kernel_ifdrops_delta": 0
}
"2019-02-19T09:30:35.000346-0600"
{
"kernel_packets": 17342905685,
"kernel_packets_delta": -4294369072,
"kernel_drops": 580833306,
"kernel_drops_delta": 0,
"kernel_ifdrops": 0,
"kernel_ifdrops_delta": 0
}
"2019-02-19T09:25:05.000338-0600"
{
"kernel_packets": 23570036423,
"kernel_packets_delta": -4293688281,
"kernel_drops": 775213362,
"kernel_drops_delta": 0,
"kernel_ifdrops": 0,
"kernel_ifdrops_delta": 0
}
"2019-02-19T08:51:53.000331-0600"
{
"kernel_packets": 12005768232,
"kernel_packets_delta": -4294159125,
"kernel_drops": 4547641950,
"kernel_drops_delta": 0,
"kernel_ifdrops": 0,
"kernel_ifdrops_delta": 0
}
"2019-02-19T08:51:03.000358-0600"
{
"kernel_packets": 22256188092,
"kernel_packets_delta": -4294023378,
"kernel_drops": 722622375,
"kernel_drops_delta": 0,
"kernel_ifdrops": 0,
"kernel_ifdrops_delta": 0
}
I do not see any messages in the suricata.log file during this time.
Is this behavior expected and if not what additional troubleshooting would you like us to perform to assist with this issue?
Files