Project

General

Profile

Actions

Bug #3880

closed

http parsing/alerting - continue

Added by Peter Manev over 4 years ago. Updated almost 3 years ago.

Status:
Closed
Priority:
Normal
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

I am not sure if this is a serious issue - hence putting it to private till otherwise said.
The data is definitely private.

There are a couple of issue (maybe one reason).
(I can share the pcap of course - but it is strictly private :) )

It seems we get 69 anomaly alerts on 20 packet one stream pcap. The amount seems excessive.
It also seems to not to parse the traffic properly.


 rm logs/* ;time  /opt/suritest/bin/suricata  -S /dev/null  -l logs/ -k none -r /home/pevma/inthetrenches/Suricata/Bugs/http_continuation_parsing_error.pcapng  ; grep '"event_type":"alert"' logs/eve.json |perl -ne 'print "$1\n" if /\"signature\":\"(.*?)\"/'  | sort | uniq -c |sort -n -r -t " " -k 1 ; cat logs/eve.json |perl -ne 'print "$1\n" if /\"event_type\":\"(.*?)\"/' | sort | uniq -c | sort -n -r  -k 1 ; grep '"event_type":"anomaly"' logs/eve.json  | jq .anomaly.event | uniq -c | sort -rn 

[972220] 15/8/2020 -- 16:09:34 - (suricata.c:1065) <Notice> (LogVersion) -- This is Suricata version 6.0.0-dev (ac491c6e8 2020-08-07) running in USER mode
[972220] 15/8/2020 -- 16:09:35 - (flow.c:635) <Notice> (FlowInitConfig) -- flow size 328, memcap allows for 409200 flows. Per hash row in perfect conditions 6
[972220] 15/8/2020 -- 16:09:35 - (tm-threads.c:1964) <Notice> (TmThreadWaitOnThreadInit) -- all 9 packet processing threads, 4 management threads initialized, engine started.
[972230] 15/8/2020 -- 16:09:35 - (flow-manager.c:805) <Notice> (FlowManager) -- FM FM#01/0 starting. min_timeout 30s. Full hash pass in 240s
[972220] 15/8/2020 -- 16:09:35 - (suricata.c:2638) <Notice> (SuricataMainLoop) -- Signal Received.  Stopping engine.
[972220] 15/8/2020 -- 16:09:35 - (flow-manager.c:1299) <Notice> (FlowDisableFlowRecyclerThread) -- flows to progress: 1
[972221] 15/8/2020 -- 16:09:35 - (source-pcap-file.c:382) <Notice> (ReceivePcapFileThreadExitStats) -- Pcap-file module read 1 files, 20 packets, 9547 bytes

real    0m0.554s
user    0m0.380s
sys    0m0.167s
     70 anomaly
      2 http
      2 fileinfo
      1 stats
      1 flow
     49 "REQUEST_BODY_UNEXPECTED" 
     20 "REQUEST_BODY_UNEXPECTED" 
      1 "UNABLE_TO_MATCH_RESPONSE_TO_REQUEST" 

grep '"event_type":"http"' logs/eve.json | jq . |more
{
  "timestamp": "2020-06-30T07:23:05.652209+0200",
  "flow_id": 517714489240655,
  "pcap_cnt": 12,
  "event_type": "http",
  "src_ip": "192.168.0.1",
  "src_port": 35137,
  "dest_ip": "10.10.10.1",
  "dest_port": 8001,
  "proto": "TCP",
  "tx_id": 0,
  "ether": {
    "src_mac": "90:1b:0e:a1:1a:33",
    "dest_mac": "00:1c:7f:f2:ae:d0" 
  },
  "http": {
    "hostname": "OBFUSCATED-ON-PURPOSE",
    "url": "/orssv/sbt_catalog/STAMUS1_0fv41eup_1_1/metadata.xml?format=xml&x-BackupType=ArchivedLog&x-BlockSize=262144&x-Chunks=1&x-CloseTime=2020-06-30%2005%3A23%3A05&x-CmpVersion=12.2.0.0.0&x-Compressed=FALSE
&x-CopyNumber=0&x-DbVersion=12.2.0.1.0&x-Dbid=1767827704&x-Dbname=STAMUS1&x-Encrypted=FALSE&x-FileName=STAMUS1_0fv41eup_1_1&x-FileSize=4980736&x-FileType=BackupPiece&x-Incarnation=KTVs8w9V0wwD&x-Incremental=FALS
E&x-NetTest=FALSE&x-Node=test-13-forstamus-1.example.com&x-OpenTime=2020-06-30%2005%3A23%3A05&x-PieceBlockSize=512&x-PieceNo=1&x-PrevOpEnd=2020-06-30%2005%3A23%3A05&x-ReqCnt=4&x-ReqTime=2020-06-30%2005%3A23%3A05
&x-SbtApi=sbtclose2&x-SbtOp=EndUpload&x-SbtVersion=12.2.0.2&x-SessionId=A9475EA89E86717FE0531F01220AFC86&x-SetCount=33807&x-SetStamp=1044429785&x-System=Linux%20x86%2064-bit&x-SystemId=13&x-Tag=AO_STAMUS1_202006
30072302&x-User=oracle",
    "http_content_type": "text/html",
    "http_method": "PUT",
    "protocol": "HTTP/1.1",
    "status": 401,
    "length": 147
  }
}
{
  "timestamp": "2020-06-30T07:23:05.663044+0200",
  "flow_id": 517714489240655,
  "pcap_cnt": 19,
  "event_type": "http",
  "src_ip": "192.168.0.1",
  "src_port": 35137,
  "dest_ip": "10.10.10.1",
  "dest_port": 8001,
  "proto": "TCP",
  "tx_id": 1,
  "ether": {
    "src_mac": "90:1b:0e:a1:1a:33",
    "dest_mac": "00:1c:7f:f2:ae:d0" 
  },
  "http": {
    "http_port": 0,
    "url": "/libhtp::request_uri_not_seen",
    "status": 200,
    "length": 3
  }
}

grep '"event_type":"alert"' logs/eve.json |perl -ne 'print "$1\n" if /\"signature\":\"(.*?)\"/'  | sort | uniq -c |sort -n -r -t " " -k 1 ; cat logs/eve.json |perl -ne 'print "$1\n" if /\"event_type\":\"(.*?)\"/' | sort | uniq -c | sort -n -r  -k 1 ; grep '"event_type":"anomaly"' logs/eve.json  | jq .anomaly.event | uniq -c | sort -rn 
     70 anomaly
      2 http
      2 fileinfo
      1 stats
      1 flow
     49 "REQUEST_BODY_UNEXPECTED" 
     20 "REQUEST_BODY_UNEXPECTED" 
      1 "UNABLE_TO_MATCH_RESPONSE_TO_REQUEST" 

cat logs/stats.log 
------------------------------------------------------------------------------------
Date: 8/15/2020 -- 16:09:35 (uptime: 0d, 00h 00m 00s)
------------------------------------------------------------------------------------
Counter                                       | TM Name                   | Value
------------------------------------------------------------------------------------
decoder.pkts                                  | Total                     | 20
decoder.bytes                                 | Total                     | 9547
decoder.ipv4                                  | Total                     | 20
decoder.ethernet                              | Total                     | 20
decoder.tcp                                   | Total                     | 20
decoder.avg_pkt_size                          | Total                     | 477
decoder.max_pkt_size                          | Total                     | 1514
decoder.max_mac_addrs_src                     | Total                     | 1
decoder.max_mac_addrs_dst                     | Total                     | 1
flow.tcp                                      | Total                     | 1
flow.wrk.spare_sync_avg                       | Total                     | 100
flow.wrk.spare_sync                           | Total                     | 1
tcp.sessions                                  | Total                     | 1
tcp.syn                                       | Total                     | 1
tcp.synack                                    | Total                     | 1
app_layer.flow.http                           | Total                     | 1
app_layer.tx.http                             | Total                     | 2
flow.mgr.full_hash_pass                       | Total                     | 1
flow.spare                                    | Total                     | 9900
flow.mgr.rows_maxlen                          | Total                     | 1
flow.mgr.flows_checked                        | Total                     | 1
flow.mgr.flows_notimeout                      | Total                     | 1
tcp.memuse                                    | Total                     | 4587520
tcp.reassembly_memuse                         | Total                     | 786432
flow.memuse                                   | Total                     | 7474304


Files

Actions #1

Updated by Peter Manev about 4 years ago

There is one pcap ready for the case above.
Working on a couple of other cases which seems related to the issue here - please see attached.

Actions #2

Updated by Victor Julien about 4 years ago

  • Status changed from New to Assigned
  • Assignee set to Philippe Antoine
  • Target version set to 6.0.0rc1
Actions #3

Updated by Philippe Antoine about 4 years ago

Interesting, it seems, there is :
- one PUT request with its headers and a content-length, and not the data
- one 401 response
- another PUT request with its headers and a content-length, and not the data
- a 100 response
- the data for the PUT request (the file whose size is the content-length)
- a 200 response

Plus we should limit to one the number of `REQUEST_BODY_UNEXPECTED` event per transaction

Actions #4

Updated by Philippe Antoine about 4 years ago

  • Status changed from Assigned to In Review

Gitlab PR
This is about `Expect` header

Actions #6

Updated by Philippe Antoine about 4 years ago

  • Status changed from In Review to Closed
Actions #7

Updated by Victor Julien almost 3 years ago

  • Private changed from Yes to No
Actions

Also available in: Atom PDF