Project

General

Profile

Actions

Bug #2099

closed

af_packet: In IPS mode some some packets are not copied

Added by Igor Novgorodov over 5 years ago. Updated about 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
-
Affected Versions:
Effort:
Difficulty:
Label:

Description

Kernel 4.9.24
Suricata 3.2.x or 4.0dev, does not matter.
1 test alert signature.

Testing with TCPReplay's sample PCAP: https://s3.amazonaws.com/tcpreplay-pcap-files/bigFlows.pcap
Sending 39580750 to eno50 interface (PCAP replayed 50 times), speed capped @ 1.15 Mpps

af-packet:
  - interface: eno49
    threads: 10
    cluster-id: 99
    cluster-type: cluster_flow
    defrag: no
    use-mmap: yes
    mmap-locked: yes
    tpacket-v3: yes
    ring-size: 524288
    block-size: 524288
    checksum-checks: no
    copy-mode: ips
    copy-iface: eno50

  - interface: eno50
    threads: 10
    cluster-id: 98
    cluster-type: cluster_flow
    defrag: no
    use-mmap: yes
    mmap-locked: yes
    tpacket-v3: yes
    ring-size: 524288
    block-size: 524288
    checksum-checks: no
    copy-mode: ips
    copy-iface: eno49

No drops on input in kernel or Suricata, but only 39 230 764 (349 986 packets gone) are sent to eno49 interface (checked by ethtool -S eno49 | grep tx_packets and sniffer connected to eno49).
This count is always the same for the same PCAP file.

Maybe Suricata drops some packets internally and does not copy them to output interface according to some internal logic?
Thanks in advance!

Resulting stats.log:

------------------------------------------------------------------------------------
Date: 4/23/2017 -- 12:11:42 (uptime: 0d, 00h 01m 39s)
------------------------------------------------------------------------------------
Counter                                    | TM Name                   | Value
------------------------------------------------------------------------------------
capture.kernel_packets                     | Total                     | 39580750
decoder.pkts                               | Total                     | 39580750
decoder.bytes                              | Total                     | 17770889200
decoder.ipv4                               | Total                     | 39558950
decoder.ipv6                               | Total                     | 21950
decoder.ethernet                           | Total                     | 39580750
decoder.tcp                                | Total                     | 31694700
decoder.udp                                | Total                     | 7656750
decoder.icmpv4                             | Total                     | 213300
decoder.icmpv6                             | Total                     | 1700
decoder.teredo                             | Total                     | 150
decoder.avg_pkt_size                       | Total                     | 448
decoder.max_pkt_size                       | Total                     | 1514
tcp.sessions                               | Total                     | 19918
tcp.pseudo                                 | Total                     | 2190
tcp.syn                                    | Total                     | 1598753
tcp.synack                                 | Total                     | 300291
tcp.rst                                    | Total                     | 70750
tcp.stream_depth_reached                   | Total                     | 3
tcp.reassembly_gap                         | Total                     | 2
detect.alert                               | Total                     | 86
app_layer.flow.http                        | Total                     | 2879
app_layer.tx.http                          | Total                     | 6604
app_layer.flow.tls                         | Total                     | 1448
app_layer.flow.failed_tcp                  | Total                     | 102
app_layer.flow.failed_udp                  | Total                     | 5055
flow.spare                                 | Total                     | 524288
flow_mgr.flows_checked                     | Total                     | 779
flow_mgr.flows_notimeout                   | Total                     | 779
flow_mgr.rows_checked                      | Total                     | 1048576
flow_mgr.rows_skipped                      | Total                     | 1047814
flow_mgr.rows_maxlen                       | Total                     | 2
tcp.memuse                                 | Total                     | 800000000
tcp.reassembly_memuse                      | Total                     | 12332832
flow.memuse                                | Total                     | 221576184


Files

Actions #1

Updated by Igor Novgorodov over 5 years ago

Update: If i set af_packet's copy-mode to 'tap' then no packets are lost.

Actions #2

Updated by Peter Manev over 5 years ago

We have investigated a similar problem with Eric and he has a patch that fixes our test case.

Would you be willing to try it out and confirm if it fixes the problem for you?

Actions #3

Updated by Eric Leblond over 5 years ago

Hi Igor. Can you try the attached patch (on top of git master) ? To activate the feature, you need to set drop-invalid: no in the stream section of the YAML.

Actions #4

Updated by Igor Novgorodov over 5 years ago

Thanks! i'll try the patch ASAP after i get back from vacation.

Actions #5

Updated by Andreas Herz over 5 years ago

  • Assignee set to OISF Dev
  • Target version set to TBD
Actions #6

Updated by Victor Julien over 5 years ago

  • Status changed from New to Assigned
  • Assignee changed from OISF Dev to Eric Leblond
  • Target version changed from TBD to 70
Actions #7

Updated by Igor Novgorodov over 5 years ago

Just checked out latest master from git, applied the patch.
It didn't applied cleanly, according to commit log, there was a lot of work on stream module, although only a minor fix was required.

I can confirm that patch works as expected - all packets are gone through AF_PACKET bridge node fine:

source# # tcpreplay -K --intf1=eno50 -l 50 -p 1150000 /opt/bigFlows.pcap
File Cache is enabled
Test start: 2017-05-09 18:18:49.498752 ...
Test complete: 2017-05-09 18:19:25.026170
Actual: 39580750 packets (17770889200 bytes) sent in 35.52 seconds
Rated: 500207988.2 Bps, 4001.66 Mbps, 1114103.35 pps
Flows: 40686 flows, 1145.20 fps, 39558950 flow packets, 21800 non-flow

Statistics for network device: eno50
        Successful packets:        39580750
        Failed packets:            0
        Truncated packets:         0
        Retried packets (ENOBUFS): 0
        Retried packets (EAGAIN):  0

suricata# suricata -c /etc/suricata/suricata.yaml --af-packet
[6106] 9/5/2017 -- 18:15:40 - (conf-yaml-loader.c:296) <Info> (ConfYamlParse) -- Including configuration file /etc/suricata/rules-custom.yaml at parent node rule-files.
[6106] 9/5/2017 -- 18:15:40 - (suricata.c:1100) <Notice> (LogVersion) -- This is Suricata version 4.0dev (rev f18c976)
[6106] 9/5/2017 -- 18:15:41 - (tm-threads.c:2178) <Notice> (TmThreadWaitOnThreadInit) -- all 20 packet processing threads, 4 management threads initialized, engine started.
[6106] 9/5/2017 -- 18:18:10 - (suricata.c:2728) <Notice> (SuricataMainLoop) -- Signal Received.  Stopping engine.
[6106] 9/5/2017 -- 18:18:12 - (util-device.c:283) <Notice> (LiveDeviceListClean) -- Stats for 'eno49':  pkts: 0, drop: 0 (-nan%), invalid chksum: 0
[6106] 9/5/2017 -- 18:18:12 - (util-device.c:283) <Notice> (LiveDeviceListClean) -- Stats for 'eno50':  pkts: 39580750, drop: 0 (0.00%), invalid chksum: 0

sink# # netsniff-ng --in eno50 -s
Running! Hang up with ^C!

    39580750  packets incoming (0 unread on exit)
    39580750  packets passed filter
           0  packets failed filter (out of space)
      0.0000% packet droprate
          49  sec, 751920 usec in total

Actions #8

Updated by Victor Julien about 5 years ago

  • Status changed from Assigned to Closed
  • Target version deleted (70)

Erics patch was merged during 4.0 development.

Actions

Also available in: Atom PDF