Suricata

Can you use iptables -vnL to see if each of your iptables rules received the same amount of packets?

Btw, the iptables rules you list have 2 that go to queue 4, none to 3. Also in the threading config I see:
- receive_cpu_set: cpu: [ "4-7" ] mode: "exclusive"
- verdict_cpu_set: cpu: [ "5-7" ] mode: "exclusive"
Shouldn't that last one be 4-7 too?

Yes

Chain FORWARD (policy ACCEPT 17 packets, 9080 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 11M  12G NFQUEUE    all  --  *      *       0.0.0.0/0            0.0.0.0/0           statistic mode nth every 4 NFQUEUE num 3
 11M  12G NFQUEUE    all  --  *      *       0.0.0.0/0            0.0.0.0/0           statistic mode nth every 3 NFQUEUE num 4
 11M  12G NFQUEUE    all  --  *      *       0.0.0.0/0            0.0.0.0/0           statistic mode nth every 2 NFQUEUE num 3
 11M  12G NFQUEUE    all  --  *      *       0.0.0.0/0            0.0.0.0/0           NFQUEUE num 2

Also, it is weird that packets processe en nfnetlink_queue is equally distributed.

Victor Julien wrote:

Can you use iptables -vnL to see if each of your iptables rules received the same amount of packets?

You are right, sorry for that. But the result i pasted with the queues was with the iptables distributing among 4 queues, not 3. And verdict cpu:[4-7]
I was messing around with the configuration trying to understand what is going on, and I forgot to correct that when i submitted the issue.

Victor Julien wrote:

Btw, the iptables rules you list have 2 that go to queue 4, none to 3. Also in the threading config I see:
- receive_cpu_set: cpu: [ "4-7" ] mode: "exclusive"
- verdict_cpu_set: cpu: [ "5-7" ] mode: "exclusive"
Shouldn't that last one be 4-7 too?

This time I see queue num 3 two times, is that right? Quite confusing.

Victor Julien wrote:

This time I see queue num 3 two times, is that right? Quite confusing.

I know, I am really sorry about that. Next time I will triple check the snapshot before sending.

I was observing that when I change the order of queues in iptables (Using the 4 queues, really) it didn't change the queue that saturates. But when I stop using one queue, or change the verdict_cpu_set setting the queue that saturates change.

Another thing, in this snapshot:

     $watch cat /proc/net/netfilter/nfnetlink_queue
              queue  pid                 dropped  
         1  -4281 17110 2 65535     0     0   398751  1                              
         2  13918  5736 2 65535     0     0   398751  1    
         3  -4279  5743 2 65535     0     0   398751  1    
         4  -4280 79972 2 65535 18256     0   398751  1

The queues 1, 2 and 3 stays around those values while in queue 4 packets keep being dropped.
In /var/log/messages I see this notification.

Apr 12 15:42:30 ips2 kernel: nf_conntrack: table full, dropping packet.

I did a

 echo 524288 > /proc/sys/net/ipv4/netfilter/ip_conntrack_max

And now all queues got saturated, although the order of saturation was q4, q1,q2 and q3.
Now I got this message that makes sense.

Apr 12 15:46:47 ips2 kernel: nf_queue: full at 80000 entries, dropping packets(s). Dropped: 1160456

So, the reason queues got overwhelmed is that I am flooding the net, but I don't understant one queue saturates much faster than other instead of being proportional.

Testing with tomahawk, I could not stress suricata that much so I didn't get any relevant result.

1  -4297    29 2 65535     0     0  1084077  1                                  
2  30061    29 2 65535     0     0  1084077  1    
3  -4298    38 2 65535     0     0  1084077  1    
4  -4299    42 2 65535     0     0  1084077  1

For the moment I am trying to simmulate some more real traffic until I can test suricata in a real environment.

Did you generate the traffic on the same box?

If this is the case, this is possible, that the traffic generator (or another service, device) is using the CPU. It could be interesting to use linux perf-tools to study how the CPU with the slow verdict is used.