Bug #3047
closedbyte_extract does not work in some situations
Description
Heelo. I tested byte_extract and found that it does not work in some situations. snort(2.9.12) detected all of the following situations:
Payload = |31 30 32 36 10 38 20 21 33 34 2e 11 22 33 44 55 66 2e 32 31|
The pcap file is also attached.
alert tcp any any -> any any (msg:"byte extract test 1"; byte_extract:2,0,two1,string,dec; content:"|33 34|"; offset:0; depth:two1; sid:1; rev:1;) alert tcp any any -> any any (msg:"byte extract test 2"; byte_extract:1,2,two2,string,dec; content:"|33 34|"; offset:8; depth:two2; sid:2; rev:1;) alert tcp any any -> any any (msg:"byte extract test 3"; byte_extract:1,2,two3,string,dec; byte_extract:1,5,eight,string,dec; content:"|33 34|"; offset:eight; depth:two3; sid:3; rev:1;) alert tcp any any -> any any (msg:"byte extract test 4"; byte_extract:1,3,sixd1,string,dec; content:"|31 30|"; content:"|33 34|"; distance:sixd1; sid:4; rev:1;) alert tcp any any -> any any (msg:"byte extract test 5"; byte_extract:1,2,twow,string,dec; byte_extract:1,3,sixd2,string,dec; content:"|31 30|"; content:"|33 34|"; distance:sixd2; within:twow; sid: 5; rev:1;) alert tcp any any -> any any (msg:"byte extract test 6"; content:"|31 30|"; byte_extract:1,6,three1,relative,string,dec; content:"|36 10|"; offset:three1; depth:2; sid:6; rev:1;) alert tcp any any -> any any (msg:"byte extract test 7"; byte_extract:1,2,dectwo1,string,dec; content:"|32|"; offset:dectwo1; depth:1; sid:7; rev:1;) alert tcp any any -> any any (msg:"byte extract test 8"; byte_extract:1,2,dectwo2,string,dec; content:"|32|"; offset:dectwo2; sid:8; rev:1;) alert tcp any any -> any any (msg:"byte extract test 9"; byte_extract:1,4,hexten; byte_extract:1,0,decone,string,dec; content:"|66|"; offset:hexten; depth:decone; sid:9; rev:1;) alert tcp any any -> any any (msg:"byte extract test 10"; byte_extract:1,4,two4; content:"|31|"; depth:1; content:"|2e|"; distance:two4; within:1; sid:10; rev:1;)
suricata alert: sid:1, sid:4, sid:5, sid:8
snort alert: all rules
If offset and depth are used together except "offset: 0", they are not alerted.
In case of sid: 10, it will be alerted when removing within or depth.
Is it correct not to work in suricata? Or is it a bug?
Please check the above. Thank you.
Files
Updated by Andreas Herz about 5 years ago
- Assignee set to Community Ticket
- Target version set to Support
There are some differences between suricata and snort regarding this keyword, see https://suricata.readthedocs.io/en/latest/rules/differences-from-snort.html#byte-extract-keyword.
But thanks for submitting rules AND the pcap so we could dig into that.
Updated by Victor Julien about 5 years ago
- Assignee changed from Community Ticket to Andreas Herz
Andreas, could you create a set of suricata-verify tests for this?
Updated by Andreas Herz about 5 years ago
sure, can we use those rules and this pcap or do we need dedicated permission?
I also confirmed that as soon as I replace the vars with the value it works as expected, so no issue with the keywords like distance, depth IMHO.
Updated by Andreas Herz about 5 years ago
What's also confusing to me, that most of those rules trigger twice even with runmode single. Can anyone confirm and has an idea why?
sudo /usr/bin/suricata -c /etc/suricata/suricata.yaml -S /tmp/byteextract.rules -r /tmp/byte_extract_test.pcap --runmode=single
sudo tail -f /var/log/suricata/eve.json| jq 'select(.event_type=="alert")'
{ "timestamp": "2019-06-14T17:45:12.603582+0200", "flow_id": 1614610294124411, "pcap_cnt": 4, "event_type": "alert", "src_ip": "10.0.0.76", "src_port": 5362, "dest_ip": "10.0.0.199", "dest_port": 80, "proto": "TCP", "alert": { "action": "allowed", "gid": 1, "signature_id": 1, "rev": 1, "signature": "byte extract test 1", "category": "", "severity": 3 }, "flow": { "pkts_toserver": 3, "pkts_toclient": 1, "bytes_toserver": 200, "bytes_toclient": 66, "start": "2019-06-14T17:45:12.601979+0200" } } { "timestamp": "2019-06-14T17:45:12.610278+0200", "flow_id": 1614610294124411, "event_type": "alert", "src_ip": "10.0.0.76", "src_port": 5362, "dest_ip": "10.0.0.199", "dest_port": 80, "proto": "TCP", "alert": { "action": "allowed", "gid": 1, "signature_id": 1, "rev": 1, "signature": "byte extract test 1", "category": "", "severity": 3 }, "http": { "http_port": 0, "url": "!34.\u0011\"3DUf.21", "http_content_type": "text/html", "http_method": "1026\u00108", "status": 400, "length": 182 }, "app_proto": "http", "app_proto_ts": "failed", "flow": { "pkts_toserver": 5, "pkts_toclient": 5, "bytes_toserver": 320, "bytes_toclient": 643, "start": "2019-06-14T17:45:12.601979+0200" } }
The second timestamp would match the last ACK packet but that doesn't have the content.
I can also reproduce it without byte_extract with this rule:
alert tcp any any -> any any (msg:"byte extract test 1"; content:"|33 34|"; offset:0; depth:10; sid:1; rev:1;)
Updated by Victor Julien about 5 years ago
This is probably because the rule matches on both packet payload and stream payload. --engine-analysis could be used to confirm.
Wrt the pcap/rules. Yeah you can use them in the tests. Just add a readme with a link to the ticket.
Updated by Andreas Herz about 5 years ago
- Tracker changed from Support to Bug
- Status changed from New to Assigned
- Target version changed from Support to 70
Updated by Victor Julien about 5 years ago
- Status changed from Assigned to New
- Assignee changed from Andreas Herz to OISF Dev
- Priority changed from Normal to High
- Target version changed from 70 to 5.0rc1
Updated by Victor Julien almost 5 years ago
- Target version changed from 5.0rc1 to 70
Updated by Victor Julien almost 5 years ago
Test cases: https://github.com/OISF/suricata-verify/pull/86
Updated by Jungho Yoon almost 5 years ago
In my environment it's still the same. Can you tell me the settings for the above test?
New installation for pcap reading.
suricata.yaml is the same as the default value.
[root@readids ~]# suricata --build-info This is Suricata version 5.0.0 RELEASE Features: PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_LIBJANSSON PROFILING TLS MAGIC RUST SIMD support: none Atomic intrinsics: 1 2 4 8 byte(s) 64-bits, Little-endian architecture GCC version 9.2.1 20190827 (Red Hat 9.2.1-1), C version 199901 compiled with _FORTIFY_SOURCE=0 L1 cache line size (CLS)=64 thread local storage method: __thread compiled with LibHTP v0.5.31, linked against LibHTP v0.5.31 Suricata Configuration: AF_PACKET support: yes eBPF support: no XDP support: no PF_RING support: no NFQueue support: no NFLOG support: no IPFW support: no Netmap support: no DAG enabled: no Napatech enabled: no WinDivert enabled: no Unix socket enabled: yes Detection enabled: yes Libmagic support: yes libnss support: yes libnspr support: yes libjansson support: yes hiredis support: no hiredis async with libevent: no Prelude support: no PCRE jit: yes LUA support: no libluajit: no GeoIP2 support: no Non-bundled htp: no Old barnyard2 support: no Hyperscan support: yes Libnet support: yes liblz4 support: yes Rust support: yes Rust strict mode: no Rust compiler path: /usr/bin/rustc Rust compiler version: rustc 1.38.0 Cargo path: /usr/bin/cargo Cargo version: cargo 1.38.0 Python support: yes Python path: /usr/bin/python3 Python distutils yes Python yaml yes Install suricatactl: yes Install suricatasc: yes Install suricata-update: yes Profiling enabled: yes Profiling locks enabled: no Development settings: Coccinelle / spatch: no Unit tests enabled: no Debug output enabled: no Debug validation enabled: no Generic build parameters: Installation prefix: /usr Configuration directory: /etc/suricata/ Log directory: /var/log/suricata/ --prefix /usr --sysconfdir /etc --localstatedir /var --datarootdir /usr/share Host: x86_64-pc-linux-gnu Compiler: gcc (exec name) / gcc (real) GCC Protect enabled: no GCC march native enabled: no GCC Profile enabled: no Position Independent Executable enabled: no CFLAGS -g -O2 -I${srcdir}/../rust/gen/c-headers
[root@readids ~]# suricata -c /etc/suricata/suricata.yaml -k none -r /root/pcap/byte_extract_test.pcap ; cat fast.log 22/10/2019 -- 20:39:00 - <Notice> - This is Suricata version 5.0.0 RELEASE running in USER mode 22/10/2019 -- 20:39:00 - <Warning> - [ERRCODE: SC_WARN_NO_JA3_SUPPORT(308)] - no MD5 calculation support built in (LibNSS), disabling JA3 22/10/2019 -- 20:39:00 - <Notice> - all 5 packet processing threads, 4 management threads initialized, engine started. 22/10/2019 -- 20:39:00 - <Notice> - Signal Received. Stopping engine. 22/10/2019 -- 20:39:00 - <Notice> - Pcap-file module read 1 files, 10 packets, 963 bytes 06/15/2019-00:45:12.603582 [**] [1:1:1] byte extract test 1 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.76:5362 -> 10.0.0.199:80 06/15/2019-00:45:12.603582 [**] [1:8:1] byte extract test 8 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.76:5362 -> 10.0.0.199:80 06/15/2019-00:45:12.610278 [**] [1:1:1] byte extract test 1 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.76:5362 -> 10.0.0.199:80 06/15/2019-00:45:12.610278 [**] [1:4:1] byte extract test 4 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.76:5362 -> 10.0.0.199:80 06/15/2019-00:45:12.610278 [**] [1:5:1] byte extract test 5 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.76:5362 -> 10.0.0.199:80 06/15/2019-00:45:12.610278 [**] [1:8:1] byte extract test 8 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.76:5362 -> 10.0.0.199:80 [root@readids ~]# cat suricata.log 22/10/2019 -- 20:39:00 - <Notice> - This is Suricata version 5.0.0 RELEASE running in USER mode 22/10/2019 -- 20:39:00 - <Info> - CPUs/cores online: 4 22/10/2019 -- 20:39:00 - <Warning> - [ERRCODE: SC_WARN_NO_JA3_SUPPORT(308)] - no MD5 calculation support built in (LibNSS), disabling JA3 22/10/2019 -- 20:39:00 - <Info> - fast output device (regular) initialized: fast.log 22/10/2019 -- 20:39:00 - <Info> - eve-log output device (regular) initialized: eve.json 22/10/2019 -- 20:39:00 - <Info> - stats output device (regular) initialized: stats.log 22/10/2019 -- 20:39:00 - <Info> - 1 rule files processed. 10 rules successfully loaded, 0 rules failed 22/10/2019 -- 20:39:00 - <Info> - Threshold config parsed: 0 rule(s) found 22/10/2019 -- 20:39:00 - <Info> - 10 signatures processed. 0 are IP-only rules, 10 are inspecting packet payload, 0 inspect application layer, 0 are decoder event only 22/10/2019 -- 20:39:00 - <Notice> - all 5 packet processing threads, 4 management threads initialized, engine started. 22/10/2019 -- 20:39:00 - <Info> - Starting file run for /root/pcap/byte_extract_test.pcap 22/10/2019 -- 20:39:00 - <Info> - pcap file /root/pcap/byte_extract_test.pcap end of file reached (pcap err code 0) 22/10/2019 -- 20:39:00 - <Notice> - Signal Received. Stopping engine. 22/10/2019 -- 20:39:00 - <Info> - time elapsed 0.031s 22/10/2019 -- 20:39:00 - <Notice> - Pcap-file module read 1 files, 10 packets, 963 bytes 22/10/2019 -- 20:39:00 - <Info> - Alerts: 6 22/10/2019 -- 20:39:00 - <Info> - cleaning up signature grouping structure... complete
Updated by Victor Julien almost 5 years ago
- Status changed from New to Assigned
- Assignee changed from OISF Dev to Jeff Lucovsky
Jeff, since you are also looking at byte_math, it might make sense to look at this one at the same time.
Updated by Jeff Lucovsky almost 5 years ago
- Status changed from Assigned to Closed
Fixed with https://github.com/OISF/suricata/pull/4391
Updated by Victor Julien almost 5 years ago
- Priority changed from High to Normal
- Target version changed from 70 to 5.0.1
- Label Needs backport added
Updated by Victor Julien almost 5 years ago
- Copied to Bug #3369: byte_extract does not work in some situations (4.1.x) added