Project

General

Profile

Actions

Bug #2841

closed

False positive alerts firing after upgrade suricata 3.0 -> 4.1.0 (4.1.x)

Added by Victor Julien almost 4 years ago. Updated over 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

Hello, oisf, could use your input on problem.

My corporate network has been running suricata 3.0. Recently upgraded to 4.1.0 then 4.1.1 and encountered issue.

Environment is CentOS release 7.4.1708 (Core) 3.10.0-862.14.4.el7.x86_64 on ~400 Dells
suricata version: suricata-4.1.0-ESG_1.el7.centos.x86_64, suricata-debuginfo-4.1.0-ESG_1.el7.centos.x86_64
pcre version: pcre-devel-8.32-17.el7.x86_64, pcre-8.32-17.el7.x86_64
lua version: lua-5.1.4-15.el7.x86_64

In Suricata 4.1.0 (and 4.1.1) (following recent upgrade from suricata 3.0) we noticed that under certain conditions false positive alerts are firing that should not be. In short, rules looking for HTTP packets are firing on ICMP data. It appears that the issue occurs on rules with http content modifiers where another rule in the ruleset uses an alert ip prefix and any content match. The packets must include an HTTP session followed by ICMP type packets (not that the rule should not match on the http session as the pcre does not match). I’ve also tested on suricata 4.1.2 and found that this issue is there as well. I’m not sure when the issue was introduced.

Rules in my local.rules:

alert ip any any -> any any (msg:"IP Any With Content"; content:"blahblah"; classtype:bad-unknown; priority:102; sid:5002203; rev:10;)

alert http $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"Test Rule no http"; flow:to_server,established; content:"POST"; http_method; pcre:"/\/.+\/[a-zA_Z0-9]{4}\_confirm\.php/"; classtype:bad-unknown; priority:102; sid:9999991; rev:1;)

Test output:

#> docker run -it --rm --volume $PWD/data:/data sensor -c /data/suricata.yaml -r ./data/out.pcap
7/1/2019 -- 15:25:55 - <Notice> - This is Suricata version 4.1.0 RELEASE
7/1/2019 -- 15:25:55 - <Notice> - using flow hash instead of active packets
7/1/2019 -- 15:25:55 - <Warning> - [ERRCODE: SC_ERR_DEPRECATED_CONF(274)] - deprecated 'force-md5' option found. Please use 'force-hash: [md5]' instead
7/1/2019 -- 15:25:55 - <Warning> - [ERRCODE: SC_ERR_EVENT_ENGINE(210)] - can't suppress sid 2015985, gid 1: unknown rule
7/1/2019 -- 15:25:55 - <Notice> - all 7 packet processing threads, 4 management threads initialized, engine started.
7/1/2019 -- 15:25:55 - <Notice> - Signal Received.  Stopping engine.
7/1/2019 -- 15:25:55 - <Notice> - Pcap-file module read 1 files, 72 packets, 10078 bytes
#>  cat ./data/fast.log 

01/02/2019-17:57:19.054415  [**] [1:9999991:1] Test Rule no http [**] [Classification: Potentially Bad Traffic] [Priority: 102] {ICMP} 3.20.244.244:2563 -> 10.228.203.106:0


Note that the rule which his supposed to match on an HTTP session is alerting on an ICMP packet.

Eve.json alert also has http data for an icmp protocol:

{
  "timestamp": "2019-01-02T17:57:19.054415+0000",
  "flow_id": 1426130925450708,
  "pcap_cnt": 22,
  "event_type": "alert",
  "src_ip": "<redacted>",
  "dest_ip": "<redacted>",
  "proto": "ICMP",
  "icmp_type": 3,
  "icmp_code": 10,
  "tx_id": 0,
  "alert": {
    "action": "allowed",
    "gid": 1,
    "signature_id": 9999991,
    "rev": 1,
    "signature": "Test Rule no http",
    "category": "Potentially Bad Traffic",
    "severity": 102
  },
  "http": {
    "hostname": "<redacted>",
    "url": "/em/event",
    "http_user_agent": "Wget/1.11.4 Red Hat modified",
    "http_method": "POST",
    "protocol": "HTTP/1.0",
    "length": 0
  },
  "app_proto": "http",
  "flow": {
    "pkts_toserver": 4,
    "pkts_toclient": 0,
    "bytes_toserver": 598,
    "bytes_toclient": 0,
    "start": "2019-01-02T17:56:06.194004+0000" 
  },
  "payload": "<redacted>",==",
  "payload_printable": "<redacted>",
  "stream": 0
}

Any thoughs on how to proceed on this would help.

Talked with Andreas Herz on this via oisf forum.


Files

issue_no_icmp-1.pcap (15.8 KB) issue_no_icmp-1.pcap pcap for Bug #2769 Charles Dillard, 01/15/2019 04:53 PM
issue_no_icmp.pcap (15.8 KB) issue_no_icmp.pcap Charles Dillard, 01/29/2019 06:37 PM
pic-for-oisf-2841.png (16.5 KB) pic-for-oisf-2841.png Charles Dillard, 02/22/2019 01:59 PM

Related issues 2 (0 open2 closed)

Copied from Bug #2769: False positive alerts firing after upgrade suricata 3.0 -> 4.1.0ClosedVictor JulienActions
Copied to Bug #2862: pcre related FP in HTTP inspection (4.1.x)ClosedVictor JulienActions
Actions #1

Updated by Victor Julien almost 4 years ago

  • Copied from Bug #2769: False positive alerts firing after upgrade suricata 3.0 -> 4.1.0 added
Actions #2

Updated by Charles Dillard almost 4 years ago

Victor Julien wrote:

Hello, oisf, could use your input on problem.

My corporate network has been running suricata 3.0. Recently upgraded to 4.1.0 then 4.1.1 and encountered issue.

Environment is CentOS release 7.4.1708 (Core) 3.10.0-862.14.4.el7.x86_64 on ~400 Dells
suricata version: suricata-4.1.0-ESG_1.el7.centos.x86_64, suricata-debuginfo-4.1.0-ESG_1.el7.centos.x86_64
pcre version: pcre-devel-8.32-17.el7.x86_64, pcre-8.32-17.el7.x86_64
lua version: lua-5.1.4-15.el7.x86_64

In Suricata 4.1.0 (and 4.1.1) (following recent upgrade from suricata 3.0) we noticed that under certain conditions false positive alerts are firing that should not be. In short, rules looking for HTTP packets are firing on ICMP data. It appears that the issue occurs on rules with http content modifiers where another rule in the ruleset uses an alert ip prefix and any content match. The packets must include an HTTP session followed by ICMP type packets (not that the rule should not match on the http session as the pcre does not match). I’ve also tested on suricata 4.1.2 and found that this issue is there as well. I’m not sure when the issue was introduced.

Rules in my local.rules:

[...]

Test output:

[...]
Note that the rule which his supposed to match on an HTTP session is alerting on an ICMP packet.

Eve.json alert also has http data for an icmp protocol:

[...]

Any thoughs on how to proceed on this would help.

Talked with Andreas Herz on this via oisf forum.

Here is status on changes. Our engineers cloned git repo to test fix. This produced files for ./configure and make Produced a tar.gz file. Built rpm from this tar.gz. This was applied to a test device. According to our incident response team, the problem persists. See ICMP and HTTP.

There is an attached file that shows what incident response team is seeing.

Actions #3

Updated by Victor Julien almost 4 years ago

Ok. I have no way to reproduce this without a test case. It would be very helpful if you can provide one. Privately if you prefer.

Actions #4

Updated by Charles Dillard almost 4 years ago

Victor Julien wrote:

Ok. I have no way to reproduce this without a test case. It would be very helpful if you can provide one. Privately if you prefer.

Will get you one.

Actions #5

Updated by Charles Dillard almost 4 years ago

Charles Dillard wrote:

Victor Julien wrote:

Ok. I have no way to reproduce this without a test case. It would be very helpful if you can provide one. Privately if you prefer.

Will get you one.

Can you give us more details on what you require for a "test case"? In the past we've provided pcap files and rules to test on.

Is it possible to set up a skype call with your team?

Actions #6

Updated by Victor Julien almost 4 years ago

Hi Charles, a test case is what you have provided before: pcap, rules, plus config and instructions if necessary. The test case provided showed one issue, but not the icmp issue.

Can you reproduce the issue with just the pcaps and rules in this ticket? If so please give detailed instructions, because I failed to reproduce the issue with them.

Actions #7

Updated by Charles Dillard almost 4 years ago

Victor Julien wrote:

Hi Charles, a test case is what you have provided before: pcap, rules, plus config and instructions if necessary. The test case provided showed one issue, but not the icmp issue.

Can you reproduce the issue with just the pcaps and rules in this ticket? If so please give detailed instructions, because I failed to reproduce the issue with them.

I will work on getting all that.

Actions #8

Updated by Victor Julien almost 4 years ago

  • Affected Versions 4.1.1, 4.1.2 added

Part of the issue is fixed in the 4.1.x branch now as well: https://github.com/OISF/suricata/pull/3681

Actions #9

Updated by Charles Dillard over 3 years ago

Victor Julien wrote:

Part of the issue is fixed in the 4.1.x branch now as well: https://github.com/OISF/suricata/pull/3681

Still working on getting authorization to send OISF the needed files.

Actions #10

Updated by Charles Dillard over 3 years ago

Charles Dillard wrote:

Victor Julien wrote:

Part of the issue is fixed in the 4.1.x branch now as well: https://github.com/OISF/suricata/pull/3681

Still working on getting authorization to send OISF the needed files.

Victor, hello, Hilary Curtis, who is our team lead, sent email to you asking if you or someone on your team would be available for a skype teleconference or other interactive means. Our incident response team leader would demonstrate the issue we are seeing with icmp/http alerts. This would circumvent the problem of sending out sensitive files. This is at your convenience.

Thanks for you consideration on this issue, Victor.

Actions #11

Updated by Victor Julien over 3 years ago

  • Copied to Bug #2862: pcre related FP in HTTP inspection (4.1.x) added
Actions #12

Updated by Victor Julien over 3 years ago

  • Target version changed from 4.1.3 to 70

I've split off #2862 from this ticket for the pcre issue that is fixed.

This ticket will remain open pending a test case for the icmp issue (private comms about this in progress).

Actions #13

Updated by Duane Howard over 3 years ago

fwiw, I'm seeing this behavior as well, but it seems extremely difficult to reproduce.
For some reason when I run with a pcap and just the rule that triggered I get no alerts, but when I run with my full rule-set I get an alert (on the same pcap). I'm struggling to figure out how to narrow this to an easy to reproduce test case.

$ suricata -r out2.pcap -l /tmp/surilog --runmode=single -S test.rules
<snip>
21/5/2019 -- 12:58:52 - <Info> - Alerts: 0
$ suricata -r out2.pcap -l /tmp/surilog --runmode=single
<snip>
20/5/2019 -- 20:31:56 - <Notice> - Pcap-file module read 1 files, 50 packets, 6324 bytes
20/5/2019 -- 20:31:56 - <Info> - Alerts: 1
Actions #14

Updated by Peter Manev over 3 years ago

Is it possible to narrow it down to a rules file/set of rules?

Actions #15

Updated by Victor Julien over 3 years ago

  • Status changed from Assigned to Closed
  • Target version changed from 70 to 4.1.5
Actions

Also available in: Atom PDF