Suricata

strip_whitespace appears to do nothing. Given the following rule:

frantrudeau@researchvm:~/testids/pcap/strip_whitespace_testing$ cat /var/lib/suricata/rules/local.rules 

alert http any any -> any any (msg:"strip_whitespace HTTP test(should not work but does)"; flow:established; file_data; strip_whitespace; content:"embrace holistically"; sid:3031337; rev:1;)

We shouldn't get hits, but do (FP):

frantrudeau@researchvm:~/testids/pcap/strip_whitespace_testing$ rm -rf log/* ; /opt/suricata/suricata-src/suricata-git/src/suricata -k none -l ./log -r bodytexttest.txt-http-get.pcap -S /var/lib/suricata/rules/local.rules -c /etc/suricata/suricata.5.0.x.local.yaml && sleep 5 && echo && cat log/local.fast.log && echo

[25981] 4/5/2020 -- 12:24:11 - (suricata.c:1066) <Notice> (LogVersion) -- This is Suricata version 6.0.0-dev (e5fd47dcf 2020-05-01) running in USER mode
[25996] 4/5/2020 -- 12:24:11 - (log-pcap.c:902) <Notice> (PcapLogInitRingBuffer) -- Ring buffer initialized with 0 files.
[25981] 4/5/2020 -- 12:24:11 - (tm-threads.c:1888) <Notice> (TmThreadWaitOnThreadInit) -- all 4 packet processing threads, 4 management threads initialized, engine started.
[25981] 4/5/2020 -- 12:24:11 - (suricata.c:2602) <Notice> (SuricataMainLoop) -- Signal Received.  Stopping engine.
[25995] 4/5/2020 -- 12:24:11 - (source-pcap-file.c:376) <Notice> (ReceivePcapFileThreadExitStats) -- Pcap-file module read 1 files, 17 packets, 5022 bytes

10/13/2008-07:55:36.182000  [**] [1:3031337:1] strip_whitespace HTTP test(should not work but does) [**] [Classification: (null)] [Priority: 3] {TCP} 173.37.145.84:80 -> 192.168.0.1:12900

frantrudeau@researchvm:~/testids/pcap/strip_whitespace_testing$ rm -rf log/* ; /opt/suricata/suricata-src/suricata-5.0.3/src/suricata -k none -l ./log -r bodytexttest.txt-http-get.pcap -S /var/lib/suricata/rules/local.rules -c /etc/suricata/suricata.5.0.x.local.yaml && sleep 5 && echo && cat log/local.fast.log && echo 
4/5/2020 -- 12:27:05 - <Notice> - This is Suricata version 5.0.3 RELEASE running in USER mode
4/5/2020 -- 12:27:05 - <Notice> - Ring buffer initialized with 0 files.
4/5/2020 -- 12:27:05 - <Notice> - all 4 packet processing threads, 4 management threads initialized, engine started.
4/5/2020 -- 12:27:05 - <Notice> - Signal Received.  Stopping engine.
4/5/2020 -- 12:27:05 - <Notice> - Pcap-file module read 1 files, 17 packets, 5022 bytes

10/13/2008-07:55:36.182000  [**] [1:3031337:1] strip_whitespace HTTP test(should not work but does) [**] [Classification: (null)] [Priority: 3] {TCP} 173.37.145.84:80 -> 192.168.0.1:12900

frantrudeau@researchvm:~/testids/pcap/strip_whitespace_testing$ rm -rf log/* ; /opt/suricata/suricata-src/suricata-4.1.7/src/suricata -k none -l ./log -r bodytexttest.txt-http-get.pcap -S /var/lib/suricata/rules/local.rules -c /etc/suricata/suricata.4.1.x.local.yaml && sleep 5 && echo && cat log/local.fast.log && echo  
4/5/2020 -- 12:27:56 - <Notice> - This is Suricata version 4.1.7 RELEASE
4/5/2020 -- 12:27:56 - <Notice> - Ring buffer initialized with 0 files.
4/5/2020 -- 12:27:56 - <Notice> - all 4 packet processing threads, 4 management threads initialized, engine started.
4/5/2020 -- 12:27:56 - <Notice> - Signal Received.  Stopping engine.
4/5/2020 -- 12:27:56 - <Notice> - Pcap-file module read 1 files, 17 packets, 5022 bytes

10/13/2008-07:55:36.182000  [**] [1:3031337:1] strip_whitespace HTTP test(should not work but does) [**] [Classification: (null)] [Priority: 3] {TCP} 173.37.145.84:80 -> 192.168.0.1:12900

Alternately, the following rule should hit, but doesn't (FN)

alert http any any -> any any (msg:"strip_whitespace HTTP test(should work but does NOT)"; flow:established; file_data; strip_whitespace; content:"embraceholistically"; sid:111111; rev:1;)

frantrudeau@researchvm:~/testids/pcap/strip_whitespace_testing$ rm -rf log/* ; /opt/suricata/suricata-src/suricata-git/src/suricata -k none -l ./log -r bodytexttest.txt-http-get.pcap -S /var/lib/suricata/rules/local.rules -c /etc/suricata/suricata.5.0.x.local.yaml && sleep 5 && echo && cat log/local.fast.log && echo
[27524] 4/5/2020 -- 12:34:09 - (suricata.c:1066) <Notice> (LogVersion) -- This is Suricata version 6.0.0-dev (e5fd47dcf 2020-05-01) running in USER mode
[27539] 4/5/2020 -- 12:34:09 - (log-pcap.c:902) <Notice> (PcapLogInitRingBuffer) -- Ring buffer initialized with 0 files.
[27524] 4/5/2020 -- 12:34:09 - (tm-threads.c:1888) <Notice> (TmThreadWaitOnThreadInit) -- all 4 packet processing threads, 4 management threads initialized, engine started.
[27524] 4/5/2020 -- 12:34:09 - (suricata.c:2602) <Notice> (SuricataMainLoop) -- Signal Received.  Stopping engine.
[27538] 4/5/2020 -- 12:34:09 - (source-pcap-file.c:376) <Notice> (ReceivePcapFileThreadExitStats) -- Pcap-file module read 1 files, 17 packets, 5022 bytes

frantrudeau@researchvm:~/testids/pcap/strip_whitespace_testing$ rm -rf log/* ; /opt/suricata/suricata-src/suricata-5.0.3/src/suricata -k none -l ./log -r bodytexttest.txt-http-get.pcap -S /var/lib/suricata/rules/local.rules -c /etc/suricata/suricata.5.0.x.local.yaml && sleep 5 && echo && cat log/local.fast.log && echo 
4/5/2020 -- 12:34:40 - <Notice> - This is Suricata version 5.0.3 RELEASE running in USER mode
4/5/2020 -- 12:34:40 - <Notice> - Ring buffer initialized with 0 files.
4/5/2020 -- 12:34:40 - <Notice> - all 4 packet processing threads, 4 management threads initialized, engine started.
4/5/2020 -- 12:34:40 - <Notice> - Signal Received.  Stopping engine.
4/5/2020 -- 12:34:40 - <Notice> - Pcap-file module read 1 files, 17 packets, 5022 bytes

frantrudeau@researchvm:~/testids/pcap/strip_whitespace_testing$ rm -rf log/* ; /opt/suricata/suricata-src/suricata-4.1.7/src/suricata -k none -l ./log -r bodytexttest.txt-http-get.pcap -S /var/lib/suricata/rules/local.rules -c /etc/suricata/suricata.4.1.x.local.yaml && sleep 5 && echo && cat log/local.fast.log && echo 
4/5/2020 -- 12:34:52 - <Notice> - This is Suricata version 4.1.7 RELEASE
4/5/2020 -- 12:34:52 - <Notice> - Ring buffer initialized with 0 files.
4/5/2020 -- 12:34:52 - <Notice> - all 4 packet processing threads, 4 management threads initialized, engine started.
4/5/2020 -- 12:34:52 - <Notice> - Signal Received.  Stopping engine.
4/5/2020 -- 12:34:52 - <Notice> - Pcap-file module read 1 files, 17 packets, 5022 bytes

frantrudeau@researchvm:~/testids/pcap/strip_whitespace_testing$