Suricata

Hello team

Some rules did not work while checking some tests for tcp fastopen.
I want to get help because I may not have understood the behavior properly.

The testing method is not very different from the existing issues(Feature #1203). I just wanted to check for additional segmentation or reassembly

Test:

client ------ suricata(IPS mode) ----- server

1. The MTU of the client is 150.
2. I added http header to http_cli.py

def main(ip, port, tfo_mode, uri):
        endpoint = (ip, port)

        HTTP_REQ_SAFE = 'GET /index.php HTTP/1.1\r\nUser-Agent: Mozilla/5.0\r\nAccept: */*\r\n\r\n'
        HTTP_REQ_EVIL = 'GET /%s HTTP/1.1\r\nUser-Agent: czxt2s\r\nAccept: */*\r\nHost: 10.0.0.61\r\nConnection: Keep-Alive\r\n\r\n' % uri

3. Send stream from client to server
python2.7 tfo_http_cli.py --tfo 10.0.0.61 80 /a1a2a3a4a5a6a7a8a9a0b1b2b3b4b5b6b7b8b9b0c1c2c3c4c5c6c7c8c9c0d1d2d3d4d5d6d7d8d9d0

then, d2 of the SYN segment is finally divided and the d3 string appears in the next segment. (See pcap for details.)
Through the rules below I expected to be all detected, but it is not.

Test Rules:

alert tcp any any -> any any (msg:"tfo test1"; content:"d2"; sid:1;)
alert tcp any any -> any any (msg:"tfo test2"; content:"d3"; sid:2;)
alert tcp-pkt any any -> any any (msg:"tfo test3"; content:"d2"; sid:3;)
alert tcp-pkt any any -> any any (msg:"tfo test4"; content:"d3"; sid:4;)
alert tcp-stream any any -> any any (msg:"tfo test5"; content:"d2"; sid:5;)
alert tcp-stream any any -> any any (msg:"tfo test6"; content:"d3"; sid:6;)
alert tcp-stream any any -> any any (msg:"tfo test7"; content:"d2d3"; sid:7;)
alert tcp any any -> any any (msg:"tfo test8"; content:"d2d3"; sid:8;)
alert tcp any any -> any any (msg:"tfo test9"; http.uri; content:"d2d3"; sid:9;)
alert tcp any any -> any any (msg:"tfo test10";  content:"GET"; sid:10;)
alert tcp any any -> any any (msg:"tfo test11"; flags:S; content:"GET"; sid:11;)
alert tcp any any -> any any (msg:"tfo test12"; http.method; content:"GET"; sid:12;)
alert http any any -> any any (msg:"tfo test13"; http.method; content:"GET"; sid:13;)
alert tcp any any -> any any (msg:"tfo test14"; http.user_agent; content:"czx"; sid:14;)
alert tcp any any -> any any (msg:"tfo test15"; http.connection; content:"Keep-Alive"; sid:15;)
alert tcp any any -> any any (msg:"tfo test16"; http.host; content:"10.0.0.61"; sid:16;)

IPS mode Detecion:

02/29/2020-23:40:46.218274  [**] [1:11:0] tfo test11 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.50:56182 -> 10.0.0.61:80
02/29/2020-23:40:46.219522  [**] [1:3:0] tfo test3 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.50:56184 -> 10.0.0.61:80
02/29/2020-23:40:46.219522  [**] [1:11:0] tfo test11 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.50:56184 -> 10.0.0.61:80
02/29/2020-23:40:46.221291  [**] [1:2:0] tfo test2 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.50:56184 -> 10.0.0.61:80
02/29/2020-23:40:46.221291  [**] [1:4:0] tfo test4 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.50:56184 -> 10.0.0.61:80
02/29/2020-23:40:46.221291  [**] [1:6:0] tfo test6 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.50:56184 -> 10.0.0.61:80
02/29/2020-23:40:46.221292  [**] [1:2:0] tfo test2 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.50:56184 -> 10.0.0.61:80
02/29/2020-23:40:46.221292  [**] [1:6:0] tfo test6 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.50:56184 -> 10.0.0.61:80

Pcap Reading:
The results were not the same.
Some rules was not detected.I think the length of the packets is not enough to reassemble in reading mode.

[root@ms-suricata ~]# suricata -c /etc/suricata/suricata.yaml -k none --simulate-ips -r /root/fastopen_segmentation4.pcap ;cat fast.log
[1420] 1/3/2020 -- 00:01:31 - (suricata.c:1654) <Info> (ParseCommandLine) -- Setting IPS mode
[1420] 1/3/2020 -- 00:01:31 - (suricata.c:1083) <Notice> (LogVersion) -- This is Suricata version 5.0.2 RELEASE running in USER mode
[1420] 1/3/2020 -- 00:01:31 - (tm-threads.c:2169) <Notice> (TmThreadWaitOnThreadInit) -- all 5 packet processing threads, 4 management threads initialized, engine started.
[1420] 1/3/2020 -- 00:01:31 - (suricata.c:2916) <Notice> (SuricataMainLoop) -- Signal Received.  Stopping engine.
[1421] 1/3/2020 -- 00:01:31 - (source-pcap-file.c:372) <Notice> (ReceivePcapFileThreadExitStats) -- Pcap-file module read 1 files, 13 packets, 1118 bytes
02/29/2020-23:40:46.219131  [**] [1:3:0] tfo test3 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.50:56184 -> 10.0.0.61:80
02/29/2020-23:40:46.219131  [**] [1:11:0] tfo test11 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.50:56184 -> 10.0.0.61:80
02/29/2020-23:40:46.217535  [**] [1:11:0] tfo test11 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.50:56182 -> 10.0.0.61:80
02/29/2020-23:40:46.220763  [**] [1:2:0] tfo test2 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.50:56184 -> 10.0.0.61:80
02/29/2020-23:40:46.220763  [**] [1:4:0] tfo test4 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.50:56184 -> 10.0.0.61:80
02/29/2020-23:40:46.220763  [**] [1:6:0] tfo test6 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.50:56184 -> 10.0.0.61:80

In normal case(non fastopen. See pcap for details.), tcp-pkt or tcp-stream was not a problem for detection. Rules were also detected except "tfo test11(sid:11)".

python2.7 tfo_http_cli.py 10.0.0.61 80 ............................/a1a2a3a4a5a6a7a8a9a0b1b2b3b4b5b6b7b8b9b0c1c2c3c4c5c6c7c8c9c0d1d2d3d4d5d6d7d8d9d0
(The normal case is also divided into segments based on d2 and d3.)

The recorded app-layer test was detected the same on my PC.
(https://github.com/OISF/suricata-verify/pull/110/commits/54d289d7ea7d1d90c616ebd70a4392a6e442072f)

However, if an HTTP transaction is split into a SYN segment and the next segment, some "http. *" Keywords does not match. (such as sid:9/14/15/16).

Test result:
1. In the fastopen case, SYN segments were detected when using the per-packet option, such as tcp-pkt or tcp-flags and not detected without the per-packet option.
2. I expected the SYN segment payload to reassembly with other segments payload in fastopen. but, raw-reassembly was not working.
3. In the fastopen case, some "http.*" keyword does not match. It was not detected by segmentation. "http.method" is not detected regardless of segmentation.

Are there any restrictions on fastopen unlike normal cases?
Please check the above. Thank you.