Project

General

Profile

Actions

Bug #3616

closed

strip_whitespace causes FN

Added by Francis Trudeau over 2 years ago. Updated almost 2 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

Tested in version 6.0.0-dev (ed8f48b05 2020-04-06), 5.0.2, 4.1.7.

The following sigs do NOT fire on the attached pcap:

alert http any any -> any any (msg:"strip_whitespace HTTP test"; flow:established; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:3034444; rev:1;)
alert tcp any any -> any any (msg:"strip_whitespace TCP test"; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:3032; rev:1;)

As opposed to the following sigs that DO fire on the attached pcap:

alert http any any -> any any (msg:"NO strip_whitespace HTTP test"; flow:established; file_data; content:"bricks-and-clicks"; sid:3033333; rev:1;)
alert tcp any any -> any any (msg:"NO strip_whitespace TCP test"; file_data; content:"bricks-and-clicks"; sid:3031; rev:1;)

This is also broken for SMTP. I can attach pcaps and sigs if needed.


Files

bodytexttest.txt-http-get.pcap (5.19 KB) bodytexttest.txt-http-get.pcap Francis Trudeau, 04/07/2020 12:18 AM

Related issues 1 (0 open1 closed)

Related to Bug #3691: strip_whitespace doesn't strip_whitespaceClosedJeff LucovskyActions
Actions #1

Updated by Peter Manev over 2 years ago

What if - if you try it with the sticky buffer "file.data" ?
Would the results be the same ?

Actions #2

Updated by Victor Julien over 2 years ago

file.data and file_data offer exactly the same functionality

Actions #3

Updated by Peter Manev over 2 years ago

Yes, was thinking if it would make a diff - however - either way it alerts on my local tests -

cat test.rules 
alert http any any -> any any (msg:"strip_whitespace HTTP test"; flow:established; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:3034444; rev:1;)
alert tcp any any -> any any (msg:"strip_whitespace TCP test"; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:3032; rev:1;)

alert http any any -> any any (msg:"strip_whitespace HTTP test"; flow:established; file.data; strip_whitespace; content:"bricks-and-clicks"; sid:6034444; rev:1;)
alert tcp any any -> any any (msg:"strip_whitespace TCP test"; file.data; strip_whitespace; content:"bricks-and-clicks"; sid:6032; rev:1;)

#alert http any any -> any any (msg:"NO strip_whitespace HTTP test"; flow:established; file_data; content:"bricks-and-clicks"; sid:3033333; rev:1;)
#alert tcp any any -> any any (msg:"NO strip_whitespace TCP test"; file_data; content:"bricks-and-clicks"; sid:3031; rev:1;)

rm log/* ;  /opt/suritest/bin/suricata -S test.rules -k none -l log/ -r /home/pevma/Downloads/bodytexttest.txt-http-get.pcap ; jq 'select (.event_type == "alert" ) ' log/eve.json | jq .alert.signature_id
[779652] 7/4/2020 -- 09:47:08 - (suricata.c:1070) <Notice> (LogVersion) -- This is Suricata version 6.0.0-dev (960c52d7f 2020-04-07) running in USER mode
[779652] 7/4/2020 -- 09:47:09 - (tm-threads.c:1887) <Notice> (TmThreadWaitOnThreadInit) -- all 9 packet processing threads, 4 management threads initialized, engine started.
[779652] 7/4/2020 -- 09:47:09 - (suricata.c:2607) <Notice> (SuricataMainLoop) -- Signal Received.  Stopping engine.
[779653] 7/4/2020 -- 09:47:09 - (source-pcap-file.c:371) <Notice> (ReceivePcapFileThreadExitStats) -- Pcap-file module read 1 files, 17 packets, 5022 bytes
3032
6032
3034444
6034444

Actions #4

Updated by Francis Trudeau over 2 years ago

Peter Manev wrote in #note-3:

Yes, was thinking if it would make a diff - however - either way it alerts on my local tests -

[...]

frantrudeau@researchvm:~/testids/pcap/strip_whitespace_testing$ cat /var/lib/suricata/rules/local.rules 
alert http any any -> any any (msg:"strip_whitespace HTTP test"; flow:established; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:3034444; rev:1;)

alert tcp any any -> any any (msg:"strip_whitespace TCP test"; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:3032; rev:1;)

alert http any any -> any any (msg:"NO strip_whitespace HTTP test"; flow:established; file_data; content:"bricks-and-clicks"; sid:3033333; rev:1;)

alert tcp any any -> any any (msg:"NO strip_whitespace TCP test"; file_data; content:"bricks-and-clicks"; sid:3031; rev:1;)

alert tcp any any -> any any (msg:"CANARY TEST"; flow:established; file_data; content:"bricks-and-clicks"; sid:3; rev:1;)
frantrudeau@researchvm:~/testids/pcap/strip_whitespace_testing$ /opt/suricata/suricata-src/suricata-git/src/suricata -k none -l /tmp/log -r bodytexttest.txt-http-get.pcap -S /var/lib/suricata/rules/local.rules -c /etc/suricata/suricata.5.0.x.local.yaml 
[7198] 7/4/2020 -- 09:12:53 - (suricata.c:1071) <Notice> (LogVersion) -- This is Suricata version 6.0.0-dev (ed8f48b05 2020-04-06) running in USER mode
[7213] 7/4/2020 -- 09:12:53 - (log-pcap.c:901) <Notice> (PcapLogInitRingBuffer) -- Ring buffer initialized with 0 files.
[7198] 7/4/2020 -- 09:12:53 - (tm-threads.c:1888) <Notice> (TmThreadWaitOnThreadInit) -- all 4 packet processing threads, 4 management threads initialized, engine started.
[7198] 7/4/2020 -- 09:12:53 - (suricata.c:2607) <Notice> (SuricataMainLoop) -- Signal Received.  Stopping engine.
[7212] 7/4/2020 -- 09:12:53 - (source-pcap-file.c:376) <Notice> (ReceivePcapFileThreadExitStats) -- Pcap-file module read 1 files, 17 packets, 5022 bytes
frantrudeau@researchvm:~/testids/pcap/strip_whitespace_testing$ cat /tmp/log/local.eve.json | jq 'select (.event_type == "alert" )' | jq '.alert | "\(.signature),\(.signature_id)" '
"CANARY TEST,3" 
"NO strip_whitespace TCP test,3031" 
"NO strip_whitespace HTTP test,3033333" 

It doesn't hit here. Peter, hit me up on IRC/Jabber/etc. please.

Actions #5

Updated by Peter Manev over 2 years ago

Currently chasing it as it apparently alerts on a couple of different Buster installations but not others - not sure if ti is OS/pkg related or something else.

Actions #6

Updated by Francis Trudeau over 2 years ago

After Peter and I mucking with this we found that strip_whitespace rules work, if no other rule has file_data without strip_whitespace:

Works:

alert http any any -> any any (msg:"CANARY TEST"; file_data; strip_whitespace; content:"commonly-accepted commonly-used"; sid:4294967294; rev:1;)

alert http any any -> any any (msg:"strip_whitespace HTTP test"; flow:established; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:3034444; rev:1;)

alert tcp any any -> any any (msg:"strip_whitespace HTTP test"; flow:established; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:10001; rev:1;)

10/13/2008-07:55:36.182000  [**] [1:10001:1] strip_whitespace HTTP test [**] [Classification: (null)] [Priority: 3] {TCP} 173.37.145.84:80 -> 192.168.0.1:12900
10/13/2008-07:55:36.182000  [**] [1:3034444:1] strip_whitespace HTTP test [**] [Classification: (null)] [Priority: 3] {TCP} 173.37.145.84:80 -> 192.168.0.1:12900
10/13/2008-07:55:36.182000  [**] [1:4294967294:1] CANARY TEST [**] [Classification: (null)] [Priority: 3] {TCP} 173.37.145.84:80 -> 192.168.0.1:12900

Does not work:

alert http any any -> any any (msg:"CANARY TEST"; file_data; content:"commonly-accepted commonly-used"; sid:4294967294; rev:1;)

alert http any any -> any any (msg:"strip_whitespace HTTP test"; flow:established; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:3034444; rev:1;)

alert tcp any any -> any any (msg:"strip_whitespace HTTP test"; flow:established; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:10001; rev:1;)

10/13/2008-07:55:36.182000  [**] [1:4294967294:1] CANARY TEST [**] [Classification: (null)] [Priority: 3] {TCP} 173.37.145.84:80 -> 192.168.0.1:12900
Actions #7

Updated by Victor Julien over 2 years ago

  • Related to Bug #3691: strip_whitespace doesn't strip_whitespace added
Actions #8

Updated by Victor Julien over 2 years ago

  • Status changed from New to Assigned
  • Assignee set to Jeff Lucovsky
  • Target version set to 6.0.0beta1

Suspect this is caused by the same issue as #3691. Can you test and craft SV tests based on the rules/pcap in this ticket?

Actions #9

Updated by Victor Julien about 2 years ago

  • Target version changed from 6.0.0beta1 to 6.0.0rc1
Actions #10

Updated by Victor Julien about 2 years ago

  • Target version changed from 6.0.0rc1 to 6.0.0
Actions #11

Updated by Victor Julien about 2 years ago

  • Target version changed from 6.0.0 to 6.0.1
Actions #13

Updated by Victor Julien about 2 years ago

I would love to see a SMTP test case as well. The mechanics of how file data is inspected is very different between http and the other protocols.

Actions #14

Updated by Jeff Lucovsky almost 2 years ago

  • Status changed from Assigned to In Review
Actions #15

Updated by Jeff Lucovsky almost 2 years ago

  • Status changed from In Review to Closed
Actions

Also available in: Atom PDF