Suricata

When using eBPF/XDP capture bypass functionality in Suricata, flows in the FLOW_STATE_CAPTURE_BYPASSED state that should timeout and get cleared from the eBPF map are not getting properly cleaned up their corresponding eBPF map entries.

This results in stale entries accumulating in the eBPF bypass map.

We have a `suricata` running in a prod environment where XDP functionality is used. We began monitoring the returns of `iface-bypassed-stat` to inspect how XDP was doing in our environment after realising a big number of our stats of `flow_bypassed.local_pkts` was really high (when we could think if XDP functionality is perfect we would have this stat near 0 and only stats on `flow_bypassed.local_capture_pkts`).

Looking at the code I suspected the code here `FlowManagerHashRowTimeout` was the culprit. We decide timeout on flow here: `FlowManagerFlowTimeout` and then use `FlowBypassedTimeout` to if there was activity at the XDP layer to indeed clean bpf map or not.

I am fairly confident on saying the bug is not here, I recompiled with a version where every flow going through here would cleanup its eBPF entry if existing and bug was still occuring.

I then suspected an issue in `CheckWorkQueue`, on reading commit

commit b3599507f4eb891841417575587d690ea13fe6c0
Author: Victor Julien <victor@inliniac.net>
Date: Thu Dec 19 17:26:45 2019 +0100

We see this change:

```
Worker threads now check the timeout of flows they evaluate during lookup.
The worker will have to read the flow into cache anyway, so the added
overhead of checking the timeout value is minimal. When a flow is considered
timed out, one of 2 things happens:

- if the flow is 'owned' by the thread it is handled locally. Handling means
      checking if the flow needs 'timeout' work.

- otherwise, the flow is added to a special 'evicted' list in the flow
      bucket where it will be picked up by the flow manager.
```

This seemed to mean that that cleanup could be handled in a worker thread and not in the flow manager. I suspected bpf map cleanup handled by `FlowBypassedTimeout` was not done here. And while the fix I tried seemed to alleviate the issue, it still remained in our prod environment (Each time I cut the network coming in suricata and monitor bpf map entries).

I do not understand where those problems may be coming from, I even tried adding a patch that deleted ebpf map key from the flow in `int FlowClearMemory(Flow* f, uint8_t proto_map)` which seemes to be called in all cases on a flow cleanup but that does not solve the problem. Some stale entries still remain.

If you have some ideas what could happen I can do some tests on our prod environment and give our `stats.log` files.