Suricata

I'm going to guess that if you run in the single threaded run mode that they are all in order? (Oops, you commented on that, sorry).

This is probably because packets are logged by multiple threads, and Suricata does not implement a re-ordering queue - this may be beyond the scope of Suricata, but I have a few ideas.

The best long term solution is probably a pcap file per thread, then an external that is aware of the multiple files and can then re-order. I'm looking into this for dumpy, however Mergecap (https://www.wireshark.org/docs/wsug_html_chunked/AppToolsmergecap.html) is another option as well.

Sounds like there is no other way but to involve a different/another tool?

Reordering within a flow shouldn't happen. What is your config?

Default suricata.yaml

I can confirm that is still an issue with current master (5.0.0-dev (rev a5f1f19b2)) and can be reproduced with the pcap and the default suricata.yaml (except enabling pcap log within).

Here's what I think is happening:

- the packets that are written to pcap are the original tunnel packets. These packets are not part of a flow, so distributed round robin over the available threads
- the encapsulated packets are split off and processed separately. They are part of a flow, so handled by a single thread.
- the pcap log is written to by all threads, so the outer layer packets are written to file by all the threads. Due to timing non-determinism, they won't be in-order.
- this means the encapsulated flow is also out of order in the pcap

I can't really think of an easy solution, other than these 2:
1. force outer layer packets to go through a single thread (e.g. always the first thread). This would be a hack and not be very scalable.
2. track flow for the outer layer. This would automagically have the same effect as (1), except it will be more scalable as the flow balancing would kick in. A side effect is that flow logging would happen too and detection of IP-only rules would change.

Even without a flow, we could still use the encapsulating ippair to compute a hash to distribute to the same thread.
Where in the code does this happen ? (This being the assignation of a packet to a Flow Worker ? )