Bug #3616
closedstrip_whitespace causes FN
Description
Tested in version 6.0.0-dev (ed8f48b05 2020-04-06), 5.0.2, 4.1.7.
The following sigs do NOT fire on the attached pcap:
alert http any any -> any any (msg:"strip_whitespace HTTP test"; flow:established; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:3034444; rev:1;)
alert tcp any any -> any any (msg:"strip_whitespace TCP test"; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:3032; rev:1;)
As opposed to the following sigs that DO fire on the attached pcap:
alert http any any -> any any (msg:"NO strip_whitespace HTTP test"; flow:established; file_data; content:"bricks-and-clicks"; sid:3033333; rev:1;)
alert tcp any any -> any any (msg:"NO strip_whitespace TCP test"; file_data; content:"bricks-and-clicks"; sid:3031; rev:1;)
This is also broken for SMTP. I can attach pcaps and sigs if needed.
Files
Updated by Peter Manev over 4 years ago
What if - if you try it with the sticky buffer "file.data" ?
Would the results be the same ?
Updated by Victor Julien over 4 years ago
file.data and file_data offer exactly the same functionality
Updated by Peter Manev over 4 years ago
Yes, was thinking if it would make a diff - however - either way it alerts on my local tests -
cat test.rules alert http any any -> any any (msg:"strip_whitespace HTTP test"; flow:established; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:3034444; rev:1;) alert tcp any any -> any any (msg:"strip_whitespace TCP test"; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:3032; rev:1;) alert http any any -> any any (msg:"strip_whitespace HTTP test"; flow:established; file.data; strip_whitespace; content:"bricks-and-clicks"; sid:6034444; rev:1;) alert tcp any any -> any any (msg:"strip_whitespace TCP test"; file.data; strip_whitespace; content:"bricks-and-clicks"; sid:6032; rev:1;) #alert http any any -> any any (msg:"NO strip_whitespace HTTP test"; flow:established; file_data; content:"bricks-and-clicks"; sid:3033333; rev:1;) #alert tcp any any -> any any (msg:"NO strip_whitespace TCP test"; file_data; content:"bricks-and-clicks"; sid:3031; rev:1;) rm log/* ; /opt/suritest/bin/suricata -S test.rules -k none -l log/ -r /home/pevma/Downloads/bodytexttest.txt-http-get.pcap ; jq 'select (.event_type == "alert" ) ' log/eve.json | jq .alert.signature_id [779652] 7/4/2020 -- 09:47:08 - (suricata.c:1070) <Notice> (LogVersion) -- This is Suricata version 6.0.0-dev (960c52d7f 2020-04-07) running in USER mode [779652] 7/4/2020 -- 09:47:09 - (tm-threads.c:1887) <Notice> (TmThreadWaitOnThreadInit) -- all 9 packet processing threads, 4 management threads initialized, engine started. [779652] 7/4/2020 -- 09:47:09 - (suricata.c:2607) <Notice> (SuricataMainLoop) -- Signal Received. Stopping engine. [779653] 7/4/2020 -- 09:47:09 - (source-pcap-file.c:371) <Notice> (ReceivePcapFileThreadExitStats) -- Pcap-file module read 1 files, 17 packets, 5022 bytes 3032 6032 3034444 6034444
Updated by Francis Trudeau over 4 years ago
Peter Manev wrote in #note-3:
Yes, was thinking if it would make a diff - however - either way it alerts on my local tests -
[...]
frantrudeau@researchvm:~/testids/pcap/strip_whitespace_testing$ cat /var/lib/suricata/rules/local.rules
alert http any any -> any any (msg:"strip_whitespace HTTP test"; flow:established; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:3034444; rev:1;)
alert tcp any any -> any any (msg:"strip_whitespace TCP test"; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:3032; rev:1;)
alert http any any -> any any (msg:"NO strip_whitespace HTTP test"; flow:established; file_data; content:"bricks-and-clicks"; sid:3033333; rev:1;)
alert tcp any any -> any any (msg:"NO strip_whitespace TCP test"; file_data; content:"bricks-and-clicks"; sid:3031; rev:1;)
alert tcp any any -> any any (msg:"CANARY TEST"; flow:established; file_data; content:"bricks-and-clicks"; sid:3; rev:1;)
frantrudeau@researchvm:~/testids/pcap/strip_whitespace_testing$ /opt/suricata/suricata-src/suricata-git/src/suricata -k none -l /tmp/log -r bodytexttest.txt-http-get.pcap -S /var/lib/suricata/rules/local.rules -c /etc/suricata/suricata.5.0.x.local.yaml
[7198] 7/4/2020 -- 09:12:53 - (suricata.c:1071) <Notice> (LogVersion) -- This is Suricata version 6.0.0-dev (ed8f48b05 2020-04-06) running in USER mode
[7213] 7/4/2020 -- 09:12:53 - (log-pcap.c:901) <Notice> (PcapLogInitRingBuffer) -- Ring buffer initialized with 0 files.
[7198] 7/4/2020 -- 09:12:53 - (tm-threads.c:1888) <Notice> (TmThreadWaitOnThreadInit) -- all 4 packet processing threads, 4 management threads initialized, engine started.
[7198] 7/4/2020 -- 09:12:53 - (suricata.c:2607) <Notice> (SuricataMainLoop) -- Signal Received. Stopping engine.
[7212] 7/4/2020 -- 09:12:53 - (source-pcap-file.c:376) <Notice> (ReceivePcapFileThreadExitStats) -- Pcap-file module read 1 files, 17 packets, 5022 bytes
frantrudeau@researchvm:~/testids/pcap/strip_whitespace_testing$ cat /tmp/log/local.eve.json | jq 'select (.event_type == "alert" )' | jq '.alert | "\(.signature),\(.signature_id)" '
"CANARY TEST,3"
"NO strip_whitespace TCP test,3031"
"NO strip_whitespace HTTP test,3033333"
It doesn't hit here. Peter, hit me up on IRC/Jabber/etc. please.
Updated by Peter Manev over 4 years ago
Currently chasing it as it apparently alerts on a couple of different Buster installations but not others - not sure if ti is OS/pkg related or something else.
Updated by Francis Trudeau over 4 years ago
After Peter and I mucking with this we found that strip_whitespace rules work, if no other rule has file_data without strip_whitespace:
Works:
alert http any any -> any any (msg:"CANARY TEST"; file_data; strip_whitespace; content:"commonly-accepted commonly-used"; sid:4294967294; rev:1;)
alert http any any -> any any (msg:"strip_whitespace HTTP test"; flow:established; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:3034444; rev:1;)
alert tcp any any -> any any (msg:"strip_whitespace HTTP test"; flow:established; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:10001; rev:1;)
10/13/2008-07:55:36.182000 [**] [1:10001:1] strip_whitespace HTTP test [**] [Classification: (null)] [Priority: 3] {TCP} 173.37.145.84:80 -> 192.168.0.1:12900
10/13/2008-07:55:36.182000 [**] [1:3034444:1] strip_whitespace HTTP test [**] [Classification: (null)] [Priority: 3] {TCP} 173.37.145.84:80 -> 192.168.0.1:12900
10/13/2008-07:55:36.182000 [**] [1:4294967294:1] CANARY TEST [**] [Classification: (null)] [Priority: 3] {TCP} 173.37.145.84:80 -> 192.168.0.1:12900
Does not work:
alert http any any -> any any (msg:"CANARY TEST"; file_data; content:"commonly-accepted commonly-used"; sid:4294967294; rev:1;)
alert http any any -> any any (msg:"strip_whitespace HTTP test"; flow:established; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:3034444; rev:1;)
alert tcp any any -> any any (msg:"strip_whitespace HTTP test"; flow:established; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:10001; rev:1;)
10/13/2008-07:55:36.182000 [**] [1:4294967294:1] CANARY TEST [**] [Classification: (null)] [Priority: 3] {TCP} 173.37.145.84:80 -> 192.168.0.1:12900
Updated by Victor Julien over 4 years ago
- Related to Bug #3691: strip_whitespace doesn't strip_whitespace added
Updated by Victor Julien over 4 years ago
- Status changed from New to Assigned
- Assignee set to Jeff Lucovsky
- Target version set to 6.0.0beta1
Suspect this is caused by the same issue as #3691. Can you test and craft SV tests based on the rules/pcap in this ticket?
Updated by Victor Julien over 4 years ago
- Target version changed from 6.0.0beta1 to 6.0.0rc1
Updated by Victor Julien about 4 years ago
- Target version changed from 6.0.0rc1 to 6.0.0
Updated by Victor Julien about 4 years ago
- Target version changed from 6.0.0 to 6.0.1
Updated by Jeff Lucovsky about 4 years ago
Updated by Victor Julien about 4 years ago
I would love to see a SMTP test case as well. The mechanics of how file data is inspected is very different between http and the other protocols.
Updated by Jeff Lucovsky about 4 years ago
- Status changed from Assigned to In Review
Updated by Jeff Lucovsky almost 4 years ago
- Status changed from In Review to Closed