Project

General

Profile

Actions

Bug #3258

open

VXLAN exceeds MTU maximum

Added by xu hui about 3 years ago. Updated 10 months ago.

Status:
Feedback
Priority:
Normal
Target version:
-
Affected Versions:
Effort:
Difficulty:
Label:

Description

My Suricata is deployed on AWS because the VXLAN protocol is used to encapsulate and send traffic mirrors, which causes the MTU to exceed the maximum.When a packet with an MTU exceeding the maximum value appears in the TCP stream, Suricata cannot perform protocol parsing on subsequent normal packets.On the other hand, WireShark can parse the contents of subsequent normal length packets.This is important to me because I can't modify the MTU in a production environment, but I need to parse the contents of subsequent normal packets.

This is a sample PCAP
First, I visited the normal page (HTTP) three times;
Then, a large page (HTTP) is accessed, triggering the VXLAN MTU to exceed its maximum value.
Finally, the normal page (HTTP) was accessed 10 times.

Suricata can only parse 4 HTTP events;
Wireshark can parse 13 HTTP events;


Files

mtu_9001_vxlan.tgz (239 KB) mtu_9001_vxlan.tgz Sample PCAP xu hui, 10/17/2019 02:34 PM
mtu_9001_vxlan.tgz (239 KB) mtu_9001_vxlan.tgz xu hui, 10/18/2019 04:12 AM
vxlan-eve.tar.gz (2.33 KB) vxlan-eve.tar.gz Peter Manev, 10/18/2019 08:41 AM

Related issues 1 (1 open0 closed)

Related to Bug #3348: Possible detection issue with VXLAN parserFeedbackTiago F.Actions
Actions #1

Updated by xu hui about 3 years ago

My Suricata is deployed on AWS because the VXLAN protocol is used to encapsulate and send traffic mirrors, which causes the MTU to exceed the maximum.When a packet with an MTU exceeding the maximum value appears in the TCP stream, Suricata cannot perform protocol parsing on subsequent normal packets.On the other hand, WireShark can parse the contents of subsequent normal length packets.This is important to me because I can't modify the MTU in a production environment, but I need to parse the contents of subsequent normal packets.

This is a sample PCAP
First, I visited the normal page (HTTP) three times;
Then, a large page (HTTP) is accessed, triggering the VXLAN MTU to exceed its maximum value.
Finally, the normal page (HTTP) was accessed 10 times.

I tried using Zeek to read pcap and compare it with Suricata.

Suricata can only parse 4 HTTP events;

{"timestamp":"2019-10-15T14:52:42.683589+0000","flow_id":1337190239592826,"event_type":"http","src_ip":"172.31.26.251","src_port":43420,"dest_ip":"172.31.7.198","dest_port":8000,"proto":"TCP","tx_id":0,"http":{"hostname":"172.31.7.198","http_port":8000,"url":"\/file\/test_files","http_user_agent":"python-requests\/1.2.3 CPython\/2.7.16 Linux\/4.14.123-86.109.amzn1.x86_64","http_content_type":"text\/html","accept":"*\/*","accept_encoding":"gzip, deflate, compress","content_length":"18","content_type":"text\/html; charset=utf-8","date":"Tue, 15 Oct 2019 14:52:42 GMT","server":"Werkzeug\/0.16.0 Python\/2.7.16","http_method":"GET","protocol":"HTTP\/1.1","status":200,"length":18}}
{"timestamp":"2019-10-15T14:52:42.728883+0000","flow_id":1337190239592826,"event_type":"http","src_ip":"172.31.26.251","src_port":43420,"dest_ip":"172.31.7.198","dest_port":8000,"proto":"TCP","tx_id":1,"http":{"hostname":"172.31.7.198","http_port":8000,"url":"\/file\/test_files","http_user_agent":"python-requests\/1.2.3 CPython\/2.7.16 Linux\/4.14.123-86.109.amzn1.x86_64","http_content_type":"text\/html","accept":"*\/*","accept_encoding":"gzip, deflate, compress","content_length":"18","content_type":"text\/html; charset=utf-8","date":"Tue, 15 Oct 2019 14:52:42 GMT","server":"Werkzeug\/0.16.0 Python\/2.7.16","http_method":"GET","protocol":"HTTP\/1.1","status":200,"length":18}}
{"timestamp":"2019-10-15T14:52:42.772883+0000","flow_id":1337190239592826,"event_type":"http","src_ip":"172.31.26.251","src_port":43420,"dest_ip":"172.31.7.198","dest_port":8000,"proto":"TCP","tx_id":2,"http":{"hostname":"172.31.7.198","http_port":8000,"url":"\/file\/test_files","http_user_agent":"python-requests\/1.2.3 CPython\/2.7.16 Linux\/4.14.123-86.109.amzn1.x86_64","http_content_type":"text\/html","accept":"*\/*","accept_encoding":"gzip, deflate, compress","content_length":"18","content_type":"text\/html; charset=utf-8","date":"Tue, 15 Oct 2019 14:52:42 GMT","server":"Werkzeug\/0.16.0 Python\/2.7.16","http_method":"GET","protocol":"HTTP\/1.1","status":200,"length":18}}
{"timestamp":"2019-10-15T14:52:43.631302+0000","flow_id":1337190239592826,"event_type":"http","src_ip":"172.31.26.251","src_port":43420,"dest_ip":"172.31.7.198","dest_port":8000,"proto":"TCP","tx_id":3,"http":{"hostname":"172.31.7.198","http_port":8000,"url":"\/file\/10mb_exist_files","http_user_agent":"python-requests\/1.2.3 CPython\/2.7.16 Linux\/4.14.123-86.109.amzn1.x86_64","accept":"*\/*","accept_encoding":"gzip, deflate, compress","http_method":"GET","protocol":"HTTP\/1.1","status":200,"length":0}}

Zeek can parse 14 HTTP events;

#separator \x09
#set_separator    ,
#empty_field    (empty)
#unset_field    -
#path    http
#open    2019-10-18-03-10-01
#fields    ts    uid    id.orig_h    id.orig_p    id.resp_h    id.resp_p    trans_depth    method    host    uri    referrer    version    user_agent    origin    request_body_len    response_body_len    status_code    status_msg    info_code    info_msg    tags    username    password    proxied    orig_fuids    orig_filenames    orig_mime_types    resp_fuids    resp_filenames    resp_mime_types
#types    time    string    addr    port    addr    port    count    string    string    string    string    string    string    string    count    count    count    string    count    string    set[enum]    string    string    set[string]    vector[string]    vector[string]    vector[string]    vector[string]    vector[string]    vector[string]
1571151162.680607    CLJ3ys3t7U8gxxL5Pd    172.31.26.251    43420    172.31.7.198    8000    1    GET    172.31.7.198    /file/test_files    -    1.1    python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64    -    0    18    200    OK    -    -    (empty)    -    -    -    -    -    -    F90WEw32ZxHF5hkIo4    -    text/plain
1571151162.684488    CLJ3ys3t7U8gxxL5Pd    172.31.26.251    43420    172.31.7.198    8000    2    GET    172.31.7.198    /file/test_files    -    1.1    python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64    -    0    18    200    OK    -    -    (empty)    -    -    -    -    -    -    FnFWrVS0tv4Jdgud1    -    text/plain
1571151162.729621    CLJ3ys3t7U8gxxL5Pd    172.31.26.251    43420    172.31.7.198    8000    3    GET    172.31.7.198    /file/test_files    -    1.1    python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64    -    0    18    200    OK    -    -    (empty)    -    -    -    -    -    -    FPiBsL3iLHagWa3zXg    -    text/plain
1571151162.773603    CLJ3ys3t7U8gxxL5Pd    172.31.26.251    43420    172.31.7.198    8000    4    GET    172.31.7.198    /file/10mb_exist_files    -    1.1    python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64    -    0    0    200    OK    -    -    (empty)    -    -    -    -    -    -    -    -    -
1571151166.786897    CLJ3ys3t7U8gxxL5Pd    172.31.26.251    43420    172.31.7.198    8000    11    GET    172.31.7.198    /file/test_files    -    -    python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64    -    0    0    -    -    -    -    (empty)    -    -    -    -    -    -    -    -    -
1571151166.284421    CLJ3ys3t7U8gxxL5Pd    172.31.26.251    43420    172.31.7.198    8000    10    GET    172.31.7.198    /file/test_files    -    -    python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64    -    0    0    -    -    -    -    (empty)    -    -    -    -    -    -    -    -    -
1571151164.274312    CLJ3ys3t7U8gxxL5Pd    172.31.26.251    43420    172.31.7.198    8000    6    GET    172.31.7.198    /file/test_files    -    -    python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64    -    0    0    -    -    -    -    (empty)    -    -    -    -    -    -    -    -    -
1571151167.792037    CLJ3ys3t7U8gxxL5Pd    172.31.26.251    43420    172.31.7.198    8000    13    GET    172.31.7.198    /file/test_files    -    -    python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64    -    0    0    -    -    -    -    (empty)    -    -    -    -    -    -    -    -    -
1571151168.294632    CLJ3ys3t7U8gxxL5Pd    172.31.26.251    43420    172.31.7.198    8000    14    GET    172.31.7.198    /file/test_files    -    -    python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64    -    0    0    -    -    -    -    (empty)    -    -    -    -    -    -    -    -    -
1571151163.728525    CLJ3ys3t7U8gxxL5Pd    172.31.26.251    43420    172.31.7.198    8000    5    GET    172.31.7.198    /file/test_files    -    -    python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64    -    0    0    -    -    -    -    (empty)    -    -    -    -    -    -    -    -    -
1571151164.776899    CLJ3ys3t7U8gxxL5Pd    172.31.26.251    43420    172.31.7.198    8000    7    GET    172.31.7.198    /file/test_files    -    -    python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64    -    0    0    -    -    -    -    (empty)    -    -    -    -    -    -    -    -    -
1571151167.289474    CLJ3ys3t7U8gxxL5Pd    172.31.26.251    43420    172.31.7.198    8000    12    GET    172.31.7.198    /file/test_files    -    -    python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64    -    0    0    -    -    -    -    (empty)    -    -    -    -    -    -    -    -    -
1571151165.279438    CLJ3ys3t7U8gxxL5Pd    172.31.26.251    43420    172.31.7.198    8000    8    GET    172.31.7.198    /file/test_files    -    -    python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64    -    0    0    -    -    -    -    (empty)    -    -    -    -    -    -    -    -    -
1571151165.781951    CLJ3ys3t7U8gxxL5Pd    172.31.26.251    43420    172.31.7.198    8000    9    GET    172.31.7.198    /file/test_files    -    -    python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64    -    0    0    -    -    -    -    (empty)    -    -    -    -    -    -    -    -    -
#close    2019-10-18-03-10-01

Actions #2

Updated by Peter Manev about 3 years ago

I just run the provided pcap against a default Suricata installation (git master) - I have all 14 http events in the log.
(Attached is the complete log as well)

sudo /opt/suritest/bin/suricata -k none -r ~/pcaps/all/qa-v2/VXLAN/mtu_9001_vxlan/mtu_9001_vxlan.pcap -k none -l log/  
[22867] 18/10/2019 -- 10:33:05 - (suricata.c:1071) <Notice> (LogVersion) -- This is Suricata version 5.0.0-dev (d5ae68afc 2019-10-15) running in USER mode
[22867] 18/10/2019 -- 10:33:05 - (output-json-dns.c:530) <Warning> (JsonDnsParseVersion) -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - eve-log dns version not found, forcing it to version 2
[22867] 18/10/2019 -- 10:33:05 - (output-json-dns.c:530) <Warning> (JsonDnsParseVersion) -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - eve-log dns version not found, forcing it to version 2
[22867] 18/10/2019 -- 10:33:58 - (tm-threads.c:2144) <Notice> (TmThreadWaitOnThreadInit) -- all 9 packet processing threads, 4 management threads initialized, engine started.
[22867] 18/10/2019 -- 10:33:58 - (suricata.c:2911) <Notice> (SuricataMainLoop) -- Signal Received.  Stopping engine.
[22968] 18/10/2019 -- 10:33:59 - (source-pcap-file.c:373) <Notice> (ReceivePcapFileThreadExitStats) -- Pcap-file module read 1 files, 6112 packets, 42410716 bytes

...
...
~/Work/Suricata/QA/tmp$ jq [.event_type] log/eve.json  |grep http | wc  -l
14
~/Work/Suricata/QA/tmp$ cat log/eve.json |perl -ne 'print "$1\n" if /\"event_type\":\"(.*?)\"/' | sort | uniq -c | sort -n -r  -k 1
     14 http
      3 fileinfo
      2 flow
      1 stats

Actions #3

Updated by xu hui about 3 years ago

Peter Manev wrote:

I just run the provided pcap against a default Suricata installation (git master) - I have all 14 http events in the log.
(Attached is the complete log as well)

[...]

Thank you for your reply, I found this problem due to too large parameter configuration.

stream:
  memcap: 1gb
  checksum-validation: no      # reject wrong csums
  inline: no                  # auto will use inline mode in IPS mode, yes or no set it statically
  midstream: true
  async-oneside: true
  reassembly:
    memcap: 2gb
    # depth: 100mb                  # before
    depth: 1mb                  # now
    toserver-chunk-size: 2560
    toclient-chunk-size: 2560
    randomize-chunk-size: yes

Actions #4

Updated by Andreas Herz about 3 years ago

  • Tracker changed from Bug to Support
  • Assignee set to Community Ticket
  • Target version set to Support

So you could solve it by an updated configuration?
What values did you have before?

Actions #5

Updated by xu hui about 3 years ago

Andreas Herz wrote:

So you could solve it by an updated configuration?
What values did you have before?

I tried to fix it by configuration, but still some packets can't fully audit the HTTP data. I didn't put those packets because of privacy issues, which made me a headache.

Actions #6

Updated by Victor Julien about 3 years ago

  • Status changed from New to Feedback

Would it be possible to craft some packets using a tool like scapy, or share a pcap privately, or scrub the pcap clean?

Actions #7

Updated by xu hui about 3 years ago

Victor Julien wrote:

Would it be possible to craft some packets using a tool like scapy, or share a pcap privately, or scrub the pcap clean?

Yeah, I have modified the sensitive content, but there is no way to modify some content, I can share it internally, I can send it to you if you need it.

Yes, I modified some sensitive content, but some sensitive content I can't completely cover.I would love to share this packet internally, hoping to help solve this problem.

Actions #8

Updated by Victor Julien almost 3 years ago

  • Related to Bug #3348: Possible detection issue with VXLAN parser added
Actions #9

Updated by xu hui almost 3 years ago

update feedback:
I tried to load the MTU overload and looked in the log and found an exception

$ cat anomaly-2019-12-10.json

{"timestamp":"2019-10-15T14:52:43.623598+0000","flow_id":1515976843223418,"event_type":"anomaly","src_ip":"172.31.26.251","src_port":43420,"dest_ip":"172.31.7.198","dest_port":8000,"proto":"TCP","packet":"RQAANGR7QAD\/BpxIrB8a+6wfB8apnB9AakHX8eFw5YE=","packet_info":{"linktype":12},"anomaly":{"type":"stream","event":"stream.reassembly_seq_gap"}}
{"timestamp":"2019-10-10T01:54:01.254359+0000","flow_id":1955350688741842,"event_type":"anomaly","src_ip":"10.168.10.209","src_port":8001,"dest_ip":"10.168.3.116","dest_port":25910,"proto":"TCP","packet":"RQAANNNMQABABkPjCqgK0QqoA3QfQWU2qFifbTFO30Q=","packet_info":{"linktype":12},"anomaly":{"type":"stream","event":"stream.reassembly_seq_gap"}}

stats.log stream.reassembly_seq_gap data in the log is very large. Is this due to MTU overload?

stream.3whs_wrong_seq_wrong_ack               | Total                     | 5627
stream.closewait_fin_out_of_window            | Total                     | 172
stream.closewait_pkt_before_last_ack          | Total                     | 79
stream.est_packet_out_of_window               | Total                     | 343
stream.est_pkt_before_last_ack                | Total                     | 5387
stream.est_synack_resend_with_diff_ack        | Total                     | 6
stream.est_syn_resend_diff_seq                | Total                     | 200
stream.est_invalid_ack                        | Total                     | 1529
stream.fin_invalid_ack                        | Total                     | 157
stream.fin1_ack_wrong_seq                     | Total                     | 2
stream.fin1_fin_wrong_seq                     | Total                     | 18
stream.fin1_invalid_ack                       | Total                     | 2
stream.fin_but_no_session                     | Total                     | 93992
stream.fin_out_of_window                      | Total                     | 103
stream.rst_but_no_session                     | Total                     | 99427
stream.timewait_ack_wrong_seq                 | Total                     | 5280
stream.shutdown_syn_resend                    | Total                     | 7
stream.pkt_invalid_timestamp                  | Total                     | 907
stream.pkt_invalid_ack                        | Total                     | 1745
stream.rst_invalid_ack                        | Total                     | 57
stream.pkt_retransmission                     | Total                     | 1625
stream.reassembly_seq_gap                     | Total                     | 3875218

Actions #10

Updated by Andreas Herz 10 months ago

  • Tracker changed from Support to Bug
Actions

Also available in: Atom PDF