Bug #3880
closedhttp parsing/alerting - continue
Description
I am not sure if this is a serious issue - hence putting it to private till otherwise said.
The data is definitely private.
There are a couple of issue (maybe one reason).
(I can share the pcap of course - but it is strictly private :) )
It seems we get 69 anomaly alerts on 20 packet one stream pcap. The amount seems excessive.
It also seems to not to parse the traffic properly.
rm logs/* ;time /opt/suritest/bin/suricata -S /dev/null -l logs/ -k none -r /home/pevma/inthetrenches/Suricata/Bugs/http_continuation_parsing_error.pcapng ; grep '"event_type":"alert"' logs/eve.json |perl -ne 'print "$1\n" if /\"signature\":\"(.*?)\"/' | sort | uniq -c |sort -n -r -t " " -k 1 ; cat logs/eve.json |perl -ne 'print "$1\n" if /\"event_type\":\"(.*?)\"/' | sort | uniq -c | sort -n -r -k 1 ; grep '"event_type":"anomaly"' logs/eve.json | jq .anomaly.event | uniq -c | sort -rn [972220] 15/8/2020 -- 16:09:34 - (suricata.c:1065) <Notice> (LogVersion) -- This is Suricata version 6.0.0-dev (ac491c6e8 2020-08-07) running in USER mode [972220] 15/8/2020 -- 16:09:35 - (flow.c:635) <Notice> (FlowInitConfig) -- flow size 328, memcap allows for 409200 flows. Per hash row in perfect conditions 6 [972220] 15/8/2020 -- 16:09:35 - (tm-threads.c:1964) <Notice> (TmThreadWaitOnThreadInit) -- all 9 packet processing threads, 4 management threads initialized, engine started. [972230] 15/8/2020 -- 16:09:35 - (flow-manager.c:805) <Notice> (FlowManager) -- FM FM#01/0 starting. min_timeout 30s. Full hash pass in 240s [972220] 15/8/2020 -- 16:09:35 - (suricata.c:2638) <Notice> (SuricataMainLoop) -- Signal Received. Stopping engine. [972220] 15/8/2020 -- 16:09:35 - (flow-manager.c:1299) <Notice> (FlowDisableFlowRecyclerThread) -- flows to progress: 1 [972221] 15/8/2020 -- 16:09:35 - (source-pcap-file.c:382) <Notice> (ReceivePcapFileThreadExitStats) -- Pcap-file module read 1 files, 20 packets, 9547 bytes real 0m0.554s user 0m0.380s sys 0m0.167s 70 anomaly 2 http 2 fileinfo 1 stats 1 flow 49 "REQUEST_BODY_UNEXPECTED" 20 "REQUEST_BODY_UNEXPECTED" 1 "UNABLE_TO_MATCH_RESPONSE_TO_REQUEST" grep '"event_type":"http"' logs/eve.json | jq . |more { "timestamp": "2020-06-30T07:23:05.652209+0200", "flow_id": 517714489240655, "pcap_cnt": 12, "event_type": "http", "src_ip": "192.168.0.1", "src_port": 35137, "dest_ip": "10.10.10.1", "dest_port": 8001, "proto": "TCP", "tx_id": 0, "ether": { "src_mac": "90:1b:0e:a1:1a:33", "dest_mac": "00:1c:7f:f2:ae:d0" }, "http": { "hostname": "OBFUSCATED-ON-PURPOSE", "url": "/orssv/sbt_catalog/STAMUS1_0fv41eup_1_1/metadata.xml?format=xml&x-BackupType=ArchivedLog&x-BlockSize=262144&x-Chunks=1&x-CloseTime=2020-06-30%2005%3A23%3A05&x-CmpVersion=12.2.0.0.0&x-Compressed=FALSE &x-CopyNumber=0&x-DbVersion=12.2.0.1.0&x-Dbid=1767827704&x-Dbname=STAMUS1&x-Encrypted=FALSE&x-FileName=STAMUS1_0fv41eup_1_1&x-FileSize=4980736&x-FileType=BackupPiece&x-Incarnation=KTVs8w9V0wwD&x-Incremental=FALS E&x-NetTest=FALSE&x-Node=test-13-forstamus-1.example.com&x-OpenTime=2020-06-30%2005%3A23%3A05&x-PieceBlockSize=512&x-PieceNo=1&x-PrevOpEnd=2020-06-30%2005%3A23%3A05&x-ReqCnt=4&x-ReqTime=2020-06-30%2005%3A23%3A05 &x-SbtApi=sbtclose2&x-SbtOp=EndUpload&x-SbtVersion=12.2.0.2&x-SessionId=A9475EA89E86717FE0531F01220AFC86&x-SetCount=33807&x-SetStamp=1044429785&x-System=Linux%20x86%2064-bit&x-SystemId=13&x-Tag=AO_STAMUS1_202006 30072302&x-User=oracle", "http_content_type": "text/html", "http_method": "PUT", "protocol": "HTTP/1.1", "status": 401, "length": 147 } } { "timestamp": "2020-06-30T07:23:05.663044+0200", "flow_id": 517714489240655, "pcap_cnt": 19, "event_type": "http", "src_ip": "192.168.0.1", "src_port": 35137, "dest_ip": "10.10.10.1", "dest_port": 8001, "proto": "TCP", "tx_id": 1, "ether": { "src_mac": "90:1b:0e:a1:1a:33", "dest_mac": "00:1c:7f:f2:ae:d0" }, "http": { "http_port": 0, "url": "/libhtp::request_uri_not_seen", "status": 200, "length": 3 } } grep '"event_type":"alert"' logs/eve.json |perl -ne 'print "$1\n" if /\"signature\":\"(.*?)\"/' | sort | uniq -c |sort -n -r -t " " -k 1 ; cat logs/eve.json |perl -ne 'print "$1\n" if /\"event_type\":\"(.*?)\"/' | sort | uniq -c | sort -n -r -k 1 ; grep '"event_type":"anomaly"' logs/eve.json | jq .anomaly.event | uniq -c | sort -rn 70 anomaly 2 http 2 fileinfo 1 stats 1 flow 49 "REQUEST_BODY_UNEXPECTED" 20 "REQUEST_BODY_UNEXPECTED" 1 "UNABLE_TO_MATCH_RESPONSE_TO_REQUEST" cat logs/stats.log ------------------------------------------------------------------------------------ Date: 8/15/2020 -- 16:09:35 (uptime: 0d, 00h 00m 00s) ------------------------------------------------------------------------------------ Counter | TM Name | Value ------------------------------------------------------------------------------------ decoder.pkts | Total | 20 decoder.bytes | Total | 9547 decoder.ipv4 | Total | 20 decoder.ethernet | Total | 20 decoder.tcp | Total | 20 decoder.avg_pkt_size | Total | 477 decoder.max_pkt_size | Total | 1514 decoder.max_mac_addrs_src | Total | 1 decoder.max_mac_addrs_dst | Total | 1 flow.tcp | Total | 1 flow.wrk.spare_sync_avg | Total | 100 flow.wrk.spare_sync | Total | 1 tcp.sessions | Total | 1 tcp.syn | Total | 1 tcp.synack | Total | 1 app_layer.flow.http | Total | 1 app_layer.tx.http | Total | 2 flow.mgr.full_hash_pass | Total | 1 flow.spare | Total | 9900 flow.mgr.rows_maxlen | Total | 1 flow.mgr.flows_checked | Total | 1 flow.mgr.flows_notimeout | Total | 1 tcp.memuse | Total | 4587520 tcp.reassembly_memuse | Total | 786432 flow.memuse | Total | 7474304
Files
Updated by Peter Manev over 4 years ago
There is one pcap ready for the case above.
Working on a couple of other cases which seems related to the issue here - please see attached.
Updated by Victor Julien over 4 years ago
- Status changed from New to Assigned
- Assignee set to Philippe Antoine
- Target version set to 6.0.0rc1
Updated by Philippe Antoine over 4 years ago
Interesting, it seems, there is :
- one PUT request with its headers and a content-length, and not the data
- one 401 response
- another PUT request with its headers and a content-length, and not the data
- a 100 response
- the data for the PUT request (the file whose size is the content-length)
- a 200 response
Plus we should limit to one the number of `REQUEST_BODY_UNEXPECTED` event per transaction
Updated by Philippe Antoine over 4 years ago
- Status changed from Assigned to In Review
Gitlab PR
This is about `Expect` header
Updated by Philippe Antoine over 4 years ago
Updated by Philippe Antoine over 4 years ago
- Status changed from In Review to Closed