Actions
Bug #3047
closedbyte_extract does not work in some situations
Description
Heelo. I tested byte_extract and found that it does not work in some situations. snort(2.9.12) detected all of the following situations:
Payload = |31 30 32 36 10 38 20 21 33 34 2e 11 22 33 44 55 66 2e 32 31|
The pcap file is also attached.
alert tcp any any -> any any (msg:"byte extract test 1"; byte_extract:2,0,two1,string,dec; content:"|33 34|"; offset:0; depth:two1; sid:1; rev:1;) alert tcp any any -> any any (msg:"byte extract test 2"; byte_extract:1,2,two2,string,dec; content:"|33 34|"; offset:8; depth:two2; sid:2; rev:1;) alert tcp any any -> any any (msg:"byte extract test 3"; byte_extract:1,2,two3,string,dec; byte_extract:1,5,eight,string,dec; content:"|33 34|"; offset:eight; depth:two3; sid:3; rev:1;) alert tcp any any -> any any (msg:"byte extract test 4"; byte_extract:1,3,sixd1,string,dec; content:"|31 30|"; content:"|33 34|"; distance:sixd1; sid:4; rev:1;) alert tcp any any -> any any (msg:"byte extract test 5"; byte_extract:1,2,twow,string,dec; byte_extract:1,3,sixd2,string,dec; content:"|31 30|"; content:"|33 34|"; distance:sixd2; within:twow; sid: 5; rev:1;) alert tcp any any -> any any (msg:"byte extract test 6"; content:"|31 30|"; byte_extract:1,6,three1,relative,string,dec; content:"|36 10|"; offset:three1; depth:2; sid:6; rev:1;) alert tcp any any -> any any (msg:"byte extract test 7"; byte_extract:1,2,dectwo1,string,dec; content:"|32|"; offset:dectwo1; depth:1; sid:7; rev:1;) alert tcp any any -> any any (msg:"byte extract test 8"; byte_extract:1,2,dectwo2,string,dec; content:"|32|"; offset:dectwo2; sid:8; rev:1;) alert tcp any any -> any any (msg:"byte extract test 9"; byte_extract:1,4,hexten; byte_extract:1,0,decone,string,dec; content:"|66|"; offset:hexten; depth:decone; sid:9; rev:1;) alert tcp any any -> any any (msg:"byte extract test 10"; byte_extract:1,4,two4; content:"|31|"; depth:1; content:"|2e|"; distance:two4; within:1; sid:10; rev:1;)
suricata alert: sid:1, sid:4, sid:5, sid:8
snort alert: all rules
If offset and depth are used together except "offset: 0", they are not alerted.
In case of sid: 10, it will be alerted when removing within or depth.
Is it correct not to work in suricata? Or is it a bug?
Please check the above. Thank you.
Files
Updated by Andreas Herz over 5 years ago
- Assignee set to Community Ticket
- Target version set to Support
There are some differences between suricata and snort regarding this keyword, see https://suricata.readthedocs.io/en/latest/rules/differences-from-snort.html#byte-extract-keyword.
But thanks for submitting rules AND the pcap so we could dig into that.
Updated by Victor Julien over 5 years ago
- Assignee changed from Community Ticket to Andreas Herz
Andreas, could you create a set of suricata-verify tests for this?
Updated by Andreas Herz over 5 years ago
sure, can we use those rules and this pcap or do we need dedicated permission?
I also confirmed that as soon as I replace the vars with the value it works as expected, so no issue with the keywords like distance, depth IMHO.
Updated by Andreas Herz over 5 years ago
What's also confusing to me, that most of those rules trigger twice even with runmode single. Can anyone confirm and has an idea why?
sudo /usr/bin/suricata -c /etc/suricata/suricata.yaml -S /tmp/byteextract.rules -r /tmp/byte_extract_test.pcap --runmode=single
sudo tail -f /var/log/suricata/eve.json| jq 'select(.event_type=="alert")'
{
"timestamp": "2019-06-14T17:45:12.603582+0200",
"flow_id": 1614610294124411,
"pcap_cnt": 4,
"event_type": "alert",
"src_ip": "10.0.0.76",
"src_port": 5362,
"dest_ip": "10.0.0.199",
"dest_port": 80,
"proto": "TCP",
"alert": {
"action": "allowed",
"gid": 1,
"signature_id": 1,
"rev": 1,
"signature": "byte extract test 1",
"category": "",
"severity": 3
},
"flow": {
"pkts_toserver": 3,
"pkts_toclient": 1,
"bytes_toserver": 200,
"bytes_toclient": 66,
"start": "2019-06-14T17:45:12.601979+0200"
}
}
{
"timestamp": "2019-06-14T17:45:12.610278+0200",
"flow_id": 1614610294124411,
"event_type": "alert",
"src_ip": "10.0.0.76",
"src_port": 5362,
"dest_ip": "10.0.0.199",
"dest_port": 80,
"proto": "TCP",
"alert": {
"action": "allowed",
"gid": 1,
"signature_id": 1,
"rev": 1,
"signature": "byte extract test 1",
"category": "",
"severity": 3
},
"http": {
"http_port": 0,
"url": "!34.\u0011\"3DUf.21",
"http_content_type": "text/html",
"http_method": "1026\u00108",
"status": 400,
"length": 182
},
"app_proto": "http",
"app_proto_ts": "failed",
"flow": {
"pkts_toserver": 5,
"pkts_toclient": 5,
"bytes_toserver": 320,
"bytes_toclient": 643,
"start": "2019-06-14T17:45:12.601979+0200"
}
}
The second timestamp would match the last ACK packet but that doesn't have the content.
I can also reproduce it without byte_extract with this rule:
alert tcp any any -> any any (msg:"byte extract test 1"; content:"|33 34|"; offset:0; depth:10; sid:1; rev:1;)
Updated by Victor Julien over 5 years ago
This is probably because the rule matches on both packet payload and stream payload. --engine-analysis could be used to confirm.
Wrt the pcap/rules. Yeah you can use them in the tests. Just add a readme with a link to the ticket.
Updated by Andreas Herz over 5 years ago
- Tracker changed from Support to Bug
- Status changed from New to Assigned
- Target version changed from Support to 70
Updated by Victor Julien over 5 years ago
- Status changed from Assigned to New
- Assignee changed from Andreas Herz to OISF Dev
- Priority changed from Normal to High
- Target version changed from 70 to 5.0rc1
Updated by Victor Julien about 5 years ago
- Target version changed from 5.0rc1 to 70
Updated by Victor Julien about 5 years ago
Test cases: https://github.com/OISF/suricata-verify/pull/86
Updated by Jungho Yoon about 5 years ago
In my environment it's still the same. Can you tell me the settings for the above test?
New installation for pcap reading.
suricata.yaml is the same as the default value.
[root@readids ~]# suricata --build-info
This is Suricata version 5.0.0 RELEASE
Features: PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_LIBJANSSON PROFILING TLS MAGIC RUST
SIMD support: none
Atomic intrinsics: 1 2 4 8 byte(s)
64-bits, Little-endian architecture
GCC version 9.2.1 20190827 (Red Hat 9.2.1-1), C version 199901
compiled with _FORTIFY_SOURCE=0
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.31, linked against LibHTP v0.5.31
Suricata Configuration:
AF_PACKET support: yes
eBPF support: no
XDP support: no
PF_RING support: no
NFQueue support: no
NFLOG support: no
IPFW support: no
Netmap support: no
DAG enabled: no
Napatech enabled: no
WinDivert enabled: no
Unix socket enabled: yes
Detection enabled: yes
Libmagic support: yes
libnss support: yes
libnspr support: yes
libjansson support: yes
hiredis support: no
hiredis async with libevent: no
Prelude support: no
PCRE jit: yes
LUA support: no
libluajit: no
GeoIP2 support: no
Non-bundled htp: no
Old barnyard2 support: no
Hyperscan support: yes
Libnet support: yes
liblz4 support: yes
Rust support: yes
Rust strict mode: no
Rust compiler path: /usr/bin/rustc
Rust compiler version: rustc 1.38.0
Cargo path: /usr/bin/cargo
Cargo version: cargo 1.38.0
Python support: yes
Python path: /usr/bin/python3
Python distutils yes
Python yaml yes
Install suricatactl: yes
Install suricatasc: yes
Install suricata-update: yes
Profiling enabled: yes
Profiling locks enabled: no
Development settings:
Coccinelle / spatch: no
Unit tests enabled: no
Debug output enabled: no
Debug validation enabled: no
Generic build parameters:
Installation prefix: /usr
Configuration directory: /etc/suricata/
Log directory: /var/log/suricata/
--prefix /usr
--sysconfdir /etc
--localstatedir /var
--datarootdir /usr/share
Host: x86_64-pc-linux-gnu
Compiler: gcc (exec name) / gcc (real)
GCC Protect enabled: no
GCC march native enabled: no
GCC Profile enabled: no
Position Independent Executable enabled: no
CFLAGS -g -O2 -I${srcdir}/../rust/gen/c-headers
[root@readids ~]# suricata -c /etc/suricata/suricata.yaml -k none -r /root/pcap/byte_extract_test.pcap ; cat fast.log
22/10/2019 -- 20:39:00 - <Notice> - This is Suricata version 5.0.0 RELEASE running in USER mode
22/10/2019 -- 20:39:00 - <Warning> - [ERRCODE: SC_WARN_NO_JA3_SUPPORT(308)] - no MD5 calculation support built in (LibNSS), disabling JA3
22/10/2019 -- 20:39:00 - <Notice> - all 5 packet processing threads, 4 management threads initialized, engine started.
22/10/2019 -- 20:39:00 - <Notice> - Signal Received. Stopping engine.
22/10/2019 -- 20:39:00 - <Notice> - Pcap-file module read 1 files, 10 packets, 963 bytes
06/15/2019-00:45:12.603582 [**] [1:1:1] byte extract test 1 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.76:5362 -> 10.0.0.199:80
06/15/2019-00:45:12.603582 [**] [1:8:1] byte extract test 8 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.76:5362 -> 10.0.0.199:80
06/15/2019-00:45:12.610278 [**] [1:1:1] byte extract test 1 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.76:5362 -> 10.0.0.199:80
06/15/2019-00:45:12.610278 [**] [1:4:1] byte extract test 4 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.76:5362 -> 10.0.0.199:80
06/15/2019-00:45:12.610278 [**] [1:5:1] byte extract test 5 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.76:5362 -> 10.0.0.199:80
06/15/2019-00:45:12.610278 [**] [1:8:1] byte extract test 8 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.76:5362 -> 10.0.0.199:80
[root@readids ~]# cat suricata.log
22/10/2019 -- 20:39:00 - <Notice> - This is Suricata version 5.0.0 RELEASE running in USER mode
22/10/2019 -- 20:39:00 - <Info> - CPUs/cores online: 4
22/10/2019 -- 20:39:00 - <Warning> - [ERRCODE: SC_WARN_NO_JA3_SUPPORT(308)] - no MD5 calculation support built in (LibNSS), disabling JA3
22/10/2019 -- 20:39:00 - <Info> - fast output device (regular) initialized: fast.log
22/10/2019 -- 20:39:00 - <Info> - eve-log output device (regular) initialized: eve.json
22/10/2019 -- 20:39:00 - <Info> - stats output device (regular) initialized: stats.log
22/10/2019 -- 20:39:00 - <Info> - 1 rule files processed. 10 rules successfully loaded, 0 rules failed
22/10/2019 -- 20:39:00 - <Info> - Threshold config parsed: 0 rule(s) found
22/10/2019 -- 20:39:00 - <Info> - 10 signatures processed. 0 are IP-only rules, 10 are inspecting packet payload, 0 inspect application layer, 0 are decoder event only
22/10/2019 -- 20:39:00 - <Notice> - all 5 packet processing threads, 4 management threads initialized, engine started.
22/10/2019 -- 20:39:00 - <Info> - Starting file run for /root/pcap/byte_extract_test.pcap
22/10/2019 -- 20:39:00 - <Info> - pcap file /root/pcap/byte_extract_test.pcap end of file reached (pcap err code 0)
22/10/2019 -- 20:39:00 - <Notice> - Signal Received. Stopping engine.
22/10/2019 -- 20:39:00 - <Info> - time elapsed 0.031s
22/10/2019 -- 20:39:00 - <Notice> - Pcap-file module read 1 files, 10 packets, 963 bytes
22/10/2019 -- 20:39:00 - <Info> - Alerts: 6
22/10/2019 -- 20:39:00 - <Info> - cleaning up signature grouping structure... complete
Updated by Victor Julien about 5 years ago
- Status changed from New to Assigned
- Assignee changed from OISF Dev to Jeff Lucovsky
Jeff, since you are also looking at byte_math, it might make sense to look at this one at the same time.
Updated by Jeff Lucovsky about 5 years ago
- Status changed from Assigned to Closed
Fixed with https://github.com/OISF/suricata/pull/4391
Updated by Victor Julien about 5 years ago
- Priority changed from High to Normal
- Target version changed from 70 to 5.0.1
- Label Needs backport added
Updated by Victor Julien about 5 years ago
- Copied to Bug #3369: byte_extract does not work in some situations (4.1.x) added
Actions