Bug #3258
openVXLAN exceeds MTU maximum
Added by xu hui about 5 years ago. Updated 5 months ago.
Description
My Suricata is deployed on AWS because the VXLAN protocol is used to encapsulate and send traffic mirrors, which causes the MTU to exceed the maximum.When a packet with an MTU exceeding the maximum value appears in the TCP stream, Suricata cannot perform protocol parsing on subsequent normal packets.On the other hand, WireShark can parse the contents of subsequent normal length packets.This is important to me because I can't modify the MTU in a production environment, but I need to parse the contents of subsequent normal packets.
This is a sample PCAP
First, I visited the normal page (HTTP) three times;
Then, a large page (HTTP) is accessed, triggering the VXLAN MTU to exceed its maximum value.
Finally, the normal page (HTTP) was accessed 10 times.
Suricata can only parse 4 HTTP events;
Wireshark can parse 13 HTTP events;
Files
| mtu_9001_vxlan.tgz (239 KB) mtu_9001_vxlan.tgz | Sample PCAP | xu hui, 10/17/2019 02:34 PM | |
| mtu_9001_vxlan.tgz (239 KB) mtu_9001_vxlan.tgz | xu hui, 10/18/2019 04:12 AM | ||
| vxlan-eve.tar.gz (2.33 KB) vxlan-eve.tar.gz | Peter Manev, 10/18/2019 08:41 AM |
Updated by xu hui about 5 years ago
- File mtu_9001_vxlan.tgz mtu_9001_vxlan.tgz added
My Suricata is deployed on AWS because the VXLAN protocol is used to encapsulate and send traffic mirrors, which causes the MTU to exceed the maximum.When a packet with an MTU exceeding the maximum value appears in the TCP stream, Suricata cannot perform protocol parsing on subsequent normal packets.On the other hand, WireShark can parse the contents of subsequent normal length packets.This is important to me because I can't modify the MTU in a production environment, but I need to parse the contents of subsequent normal packets.
This is a sample PCAP
First, I visited the normal page (HTTP) three times;
Then, a large page (HTTP) is accessed, triggering the VXLAN MTU to exceed its maximum value.
Finally, the normal page (HTTP) was accessed 10 times.
I tried using Zeek to read pcap and compare it with Suricata.
Suricata can only parse 4 HTTP events;
{"timestamp":"2019-10-15T14:52:42.683589+0000","flow_id":1337190239592826,"event_type":"http","src_ip":"172.31.26.251","src_port":43420,"dest_ip":"172.31.7.198","dest_port":8000,"proto":"TCP","tx_id":0,"http":{"hostname":"172.31.7.198","http_port":8000,"url":"\/file\/test_files","http_user_agent":"python-requests\/1.2.3 CPython\/2.7.16 Linux\/4.14.123-86.109.amzn1.x86_64","http_content_type":"text\/html","accept":"*\/*","accept_encoding":"gzip, deflate, compress","content_length":"18","content_type":"text\/html; charset=utf-8","date":"Tue, 15 Oct 2019 14:52:42 GMT","server":"Werkzeug\/0.16.0 Python\/2.7.16","http_method":"GET","protocol":"HTTP\/1.1","status":200,"length":18}}
{"timestamp":"2019-10-15T14:52:42.728883+0000","flow_id":1337190239592826,"event_type":"http","src_ip":"172.31.26.251","src_port":43420,"dest_ip":"172.31.7.198","dest_port":8000,"proto":"TCP","tx_id":1,"http":{"hostname":"172.31.7.198","http_port":8000,"url":"\/file\/test_files","http_user_agent":"python-requests\/1.2.3 CPython\/2.7.16 Linux\/4.14.123-86.109.amzn1.x86_64","http_content_type":"text\/html","accept":"*\/*","accept_encoding":"gzip, deflate, compress","content_length":"18","content_type":"text\/html; charset=utf-8","date":"Tue, 15 Oct 2019 14:52:42 GMT","server":"Werkzeug\/0.16.0 Python\/2.7.16","http_method":"GET","protocol":"HTTP\/1.1","status":200,"length":18}}
{"timestamp":"2019-10-15T14:52:42.772883+0000","flow_id":1337190239592826,"event_type":"http","src_ip":"172.31.26.251","src_port":43420,"dest_ip":"172.31.7.198","dest_port":8000,"proto":"TCP","tx_id":2,"http":{"hostname":"172.31.7.198","http_port":8000,"url":"\/file\/test_files","http_user_agent":"python-requests\/1.2.3 CPython\/2.7.16 Linux\/4.14.123-86.109.amzn1.x86_64","http_content_type":"text\/html","accept":"*\/*","accept_encoding":"gzip, deflate, compress","content_length":"18","content_type":"text\/html; charset=utf-8","date":"Tue, 15 Oct 2019 14:52:42 GMT","server":"Werkzeug\/0.16.0 Python\/2.7.16","http_method":"GET","protocol":"HTTP\/1.1","status":200,"length":18}}
{"timestamp":"2019-10-15T14:52:43.631302+0000","flow_id":1337190239592826,"event_type":"http","src_ip":"172.31.26.251","src_port":43420,"dest_ip":"172.31.7.198","dest_port":8000,"proto":"TCP","tx_id":3,"http":{"hostname":"172.31.7.198","http_port":8000,"url":"\/file\/10mb_exist_files","http_user_agent":"python-requests\/1.2.3 CPython\/2.7.16 Linux\/4.14.123-86.109.amzn1.x86_64","accept":"*\/*","accept_encoding":"gzip, deflate, compress","http_method":"GET","protocol":"HTTP\/1.1","status":200,"length":0}}
Zeek can parse 14 HTTP events;
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path http
#open 2019-10-18-03-10-01
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p trans_depth method host uri referrer version user_agent origin request_body_len response_body_len status_code status_msg info_code info_msg tags username password proxied orig_fuids orig_filenames orig_mime_types resp_fuids resp_filenames resp_mime_types
#types time string addr port addr port count string string string string string string string count count count string count string set[enum] string string set[string] vector[string] vector[string] vector[string] vector[string] vector[string] vector[string]
1571151162.680607 CLJ3ys3t7U8gxxL5Pd 172.31.26.251 43420 172.31.7.198 8000 1 GET 172.31.7.198 /file/test_files - 1.1 python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64 - 0 18 200 OK - - (empty) - - - - - - F90WEw32ZxHF5hkIo4 - text/plain
1571151162.684488 CLJ3ys3t7U8gxxL5Pd 172.31.26.251 43420 172.31.7.198 8000 2 GET 172.31.7.198 /file/test_files - 1.1 python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64 - 0 18 200 OK - - (empty) - - - - - - FnFWrVS0tv4Jdgud1 - text/plain
1571151162.729621 CLJ3ys3t7U8gxxL5Pd 172.31.26.251 43420 172.31.7.198 8000 3 GET 172.31.7.198 /file/test_files - 1.1 python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64 - 0 18 200 OK - - (empty) - - - - - - FPiBsL3iLHagWa3zXg - text/plain
1571151162.773603 CLJ3ys3t7U8gxxL5Pd 172.31.26.251 43420 172.31.7.198 8000 4 GET 172.31.7.198 /file/10mb_exist_files - 1.1 python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64 - 0 0 200 OK - - (empty) - - - - - - - - -
1571151166.786897 CLJ3ys3t7U8gxxL5Pd 172.31.26.251 43420 172.31.7.198 8000 11 GET 172.31.7.198 /file/test_files - - python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64 - 0 0 - - - - (empty) - - - - - - - - -
1571151166.284421 CLJ3ys3t7U8gxxL5Pd 172.31.26.251 43420 172.31.7.198 8000 10 GET 172.31.7.198 /file/test_files - - python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64 - 0 0 - - - - (empty) - - - - - - - - -
1571151164.274312 CLJ3ys3t7U8gxxL5Pd 172.31.26.251 43420 172.31.7.198 8000 6 GET 172.31.7.198 /file/test_files - - python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64 - 0 0 - - - - (empty) - - - - - - - - -
1571151167.792037 CLJ3ys3t7U8gxxL5Pd 172.31.26.251 43420 172.31.7.198 8000 13 GET 172.31.7.198 /file/test_files - - python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64 - 0 0 - - - - (empty) - - - - - - - - -
1571151168.294632 CLJ3ys3t7U8gxxL5Pd 172.31.26.251 43420 172.31.7.198 8000 14 GET 172.31.7.198 /file/test_files - - python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64 - 0 0 - - - - (empty) - - - - - - - - -
1571151163.728525 CLJ3ys3t7U8gxxL5Pd 172.31.26.251 43420 172.31.7.198 8000 5 GET 172.31.7.198 /file/test_files - - python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64 - 0 0 - - - - (empty) - - - - - - - - -
1571151164.776899 CLJ3ys3t7U8gxxL5Pd 172.31.26.251 43420 172.31.7.198 8000 7 GET 172.31.7.198 /file/test_files - - python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64 - 0 0 - - - - (empty) - - - - - - - - -
1571151167.289474 CLJ3ys3t7U8gxxL5Pd 172.31.26.251 43420 172.31.7.198 8000 12 GET 172.31.7.198 /file/test_files - - python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64 - 0 0 - - - - (empty) - - - - - - - - -
1571151165.279438 CLJ3ys3t7U8gxxL5Pd 172.31.26.251 43420 172.31.7.198 8000 8 GET 172.31.7.198 /file/test_files - - python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64 - 0 0 - - - - (empty) - - - - - - - - -
1571151165.781951 CLJ3ys3t7U8gxxL5Pd 172.31.26.251 43420 172.31.7.198 8000 9 GET 172.31.7.198 /file/test_files - - python-requests/1.2.3 CPython/2.7.16 Linux/4.14.123-86.109.amzn1.x86_64 - 0 0 - - - - (empty) - - - - - - - - -
#close 2019-10-18-03-10-01
Updated by Peter Manev about 5 years ago
- File vxlan-eve.tar.gz vxlan-eve.tar.gz added
I just run the provided pcap against a default Suricata installation (git master) - I have all 14 http events in the log.
(Attached is the complete log as well)
sudo /opt/suritest/bin/suricata -k none -r ~/pcaps/all/qa-v2/VXLAN/mtu_9001_vxlan/mtu_9001_vxlan.pcap -k none -l log/
[22867] 18/10/2019 -- 10:33:05 - (suricata.c:1071) <Notice> (LogVersion) -- This is Suricata version 5.0.0-dev (d5ae68afc 2019-10-15) running in USER mode
[22867] 18/10/2019 -- 10:33:05 - (output-json-dns.c:530) <Warning> (JsonDnsParseVersion) -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - eve-log dns version not found, forcing it to version 2
[22867] 18/10/2019 -- 10:33:05 - (output-json-dns.c:530) <Warning> (JsonDnsParseVersion) -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - eve-log dns version not found, forcing it to version 2
[22867] 18/10/2019 -- 10:33:58 - (tm-threads.c:2144) <Notice> (TmThreadWaitOnThreadInit) -- all 9 packet processing threads, 4 management threads initialized, engine started.
[22867] 18/10/2019 -- 10:33:58 - (suricata.c:2911) <Notice> (SuricataMainLoop) -- Signal Received. Stopping engine.
[22968] 18/10/2019 -- 10:33:59 - (source-pcap-file.c:373) <Notice> (ReceivePcapFileThreadExitStats) -- Pcap-file module read 1 files, 6112 packets, 42410716 bytes
...
...
~/Work/Suricata/QA/tmp$ jq [.event_type] log/eve.json |grep http | wc -l
14
~/Work/Suricata/QA/tmp$ cat log/eve.json |perl -ne 'print "$1\n" if /\"event_type\":\"(.*?)\"/' | sort | uniq -c | sort -n -r -k 1
14 http
3 fileinfo
2 flow
1 stats
Updated by xu hui about 5 years ago
Peter Manev wrote:
I just run the provided pcap against a default Suricata installation (git master) - I have all 14 http events in the log.
(Attached is the complete log as well)[...]
Thank you for your reply, I found this problem due to too large parameter configuration.
stream:
memcap: 1gb
checksum-validation: no # reject wrong csums
inline: no # auto will use inline mode in IPS mode, yes or no set it statically
midstream: true
async-oneside: true
reassembly:
memcap: 2gb
# depth: 100mb # before
depth: 1mb # now
toserver-chunk-size: 2560
toclient-chunk-size: 2560
randomize-chunk-size: yes
Updated by Andreas Herz about 5 years ago
- Tracker changed from Bug to Support
- Assignee set to Community Ticket
- Target version set to Support
So you could solve it by an updated configuration?
What values did you have before?
Updated by xu hui about 5 years ago
Andreas Herz wrote:
So you could solve it by an updated configuration?
What values did you have before?
I tried to fix it by configuration, but still some packets can't fully audit the HTTP data. I didn't put those packets because of privacy issues, which made me a headache.
Updated by Victor Julien about 5 years ago
- Status changed from New to Feedback
Would it be possible to craft some packets using a tool like scapy, or share a pcap privately, or scrub the pcap clean?
Updated by xu hui about 5 years ago
Victor Julien wrote:
Would it be possible to craft some packets using a tool like scapy, or share a pcap privately, or scrub the pcap clean?
Yeah, I have modified the sensitive content, but there is no way to modify some content, I can share it internally, I can send it to you if you need it.
Yes, I modified some sensitive content, but some sensitive content I can't completely cover.I would love to share this packet internally, hoping to help solve this problem.
Updated by Victor Julien almost 5 years ago
- Related to Bug #3348: Possible detection issue with VXLAN parser added
Updated by xu hui almost 5 years ago
update feedback:
I tried to load the MTU overload and looked in the log and found an exception
$ cat anomaly-2019-12-10.json
{"timestamp":"2019-10-15T14:52:43.623598+0000","flow_id":1515976843223418,"event_type":"anomaly","src_ip":"172.31.26.251","src_port":43420,"dest_ip":"172.31.7.198","dest_port":8000,"proto":"TCP","packet":"RQAANGR7QAD\/BpxIrB8a+6wfB8apnB9AakHX8eFw5YE=","packet_info":{"linktype":12},"anomaly":{"type":"stream","event":"stream.reassembly_seq_gap"}}
{"timestamp":"2019-10-10T01:54:01.254359+0000","flow_id":1955350688741842,"event_type":"anomaly","src_ip":"10.168.10.209","src_port":8001,"dest_ip":"10.168.3.116","dest_port":25910,"proto":"TCP","packet":"RQAANNNMQABABkPjCqgK0QqoA3QfQWU2qFifbTFO30Q=","packet_info":{"linktype":12},"anomaly":{"type":"stream","event":"stream.reassembly_seq_gap"}}
stats.log stream.reassembly_seq_gap data in the log is very large. Is this due to MTU overload?
stream.3whs_wrong_seq_wrong_ack | Total | 5627
stream.closewait_fin_out_of_window | Total | 172
stream.closewait_pkt_before_last_ack | Total | 79
stream.est_packet_out_of_window | Total | 343
stream.est_pkt_before_last_ack | Total | 5387
stream.est_synack_resend_with_diff_ack | Total | 6
stream.est_syn_resend_diff_seq | Total | 200
stream.est_invalid_ack | Total | 1529
stream.fin_invalid_ack | Total | 157
stream.fin1_ack_wrong_seq | Total | 2
stream.fin1_fin_wrong_seq | Total | 18
stream.fin1_invalid_ack | Total | 2
stream.fin_but_no_session | Total | 93992
stream.fin_out_of_window | Total | 103
stream.rst_but_no_session | Total | 99427
stream.timewait_ack_wrong_seq | Total | 5280
stream.shutdown_syn_resend | Total | 7
stream.pkt_invalid_timestamp | Total | 907
stream.pkt_invalid_ack | Total | 1745
stream.rst_invalid_ack | Total | 57
stream.pkt_retransmission | Total | 1625
stream.reassembly_seq_gap | Total | 3875218