Project

General

Profile

Actions

Bug #3047

closed

byte_extract does not work in some situations

Added by Jungho Yoon over 2 years ago. Updated almost 2 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:
Needs backport

Description

Heelo. I tested byte_extract and found that it does not work in some situations. snort(2.9.12) detected all of the following situations:

Payload = |31 30 32 36 10 38 20 21 33 34 2e 11 22 33 44 55 66 2e 32 31|
The pcap file is also attached.

alert tcp any any -> any any (msg:"byte extract test 1"; byte_extract:2,0,two1,string,dec; content:"|33 34|"; offset:0; depth:two1; sid:1; rev:1;)
alert tcp any any -> any any (msg:"byte extract test 2"; byte_extract:1,2,two2,string,dec; content:"|33 34|"; offset:8; depth:two2; sid:2; rev:1;)
alert tcp any any -> any any (msg:"byte extract test 3"; byte_extract:1,2,two3,string,dec; byte_extract:1,5,eight,string,dec; content:"|33 34|"; offset:eight; depth:two3; sid:3; rev:1;)
alert tcp any any -> any any (msg:"byte extract test 4"; byte_extract:1,3,sixd1,string,dec; content:"|31 30|"; content:"|33 34|"; distance:sixd1; sid:4; rev:1;)
alert tcp any any -> any any (msg:"byte extract test 5"; byte_extract:1,2,twow,string,dec; byte_extract:1,3,sixd2,string,dec; content:"|31 30|"; content:"|33 34|"; distance:sixd2; within:twow; sid: 5; rev:1;)
alert tcp any any -> any any (msg:"byte extract test 6"; content:"|31 30|"; byte_extract:1,6,three1,relative,string,dec; content:"|36 10|"; offset:three1; depth:2; sid:6; rev:1;)
alert tcp any any -> any any (msg:"byte extract test 7"; byte_extract:1,2,dectwo1,string,dec; content:"|32|"; offset:dectwo1; depth:1; sid:7; rev:1;)
alert tcp any any -> any any (msg:"byte extract test 8"; byte_extract:1,2,dectwo2,string,dec; content:"|32|"; offset:dectwo2; sid:8; rev:1;)
alert tcp any any -> any any (msg:"byte extract test 9"; byte_extract:1,4,hexten; byte_extract:1,0,decone,string,dec; content:"|66|"; offset:hexten; depth:decone; sid:9; rev:1;)
alert tcp any any -> any any (msg:"byte extract test 10"; byte_extract:1,4,two4; content:"|31|"; depth:1; content:"|2e|"; distance:two4; within:1; sid:10; rev:1;)

suricata alert: sid:1, sid:4, sid:5, sid:8
snort alert: all rules

If offset and depth are used together except "offset: 0", they are not alerted.
In case of sid: 10, it will be alerted when removing within or depth.

Is it correct not to work in suricata? Or is it a bug?

Please check the above. Thank you.


Files

byte_extract_test.pcap (1.12 KB) byte_extract_test.pcap Jungho Yoon, 06/14/2019 03:49 PM

Related issues

Copied to Bug #3369: byte_extract does not work in some situations (4.1.x)ClosedJeff LucovskyActions
Actions #1

Updated by Andreas Herz over 2 years ago

  • Assignee set to Community Ticket
  • Target version set to Support

There are some differences between suricata and snort regarding this keyword, see https://suricata.readthedocs.io/en/latest/rules/differences-from-snort.html#byte-extract-keyword.

But thanks for submitting rules AND the pcap so we could dig into that.

Actions #2

Updated by Victor Julien over 2 years ago

  • Assignee changed from Community Ticket to Andreas Herz

Andreas, could you create a set of suricata-verify tests for this?

Actions #3

Updated by Andreas Herz over 2 years ago

sure, can we use those rules and this pcap or do we need dedicated permission?

I also confirmed that as soon as I replace the vars with the value it works as expected, so no issue with the keywords like distance, depth IMHO.

Actions #4

Updated by Andreas Herz over 2 years ago

What's also confusing to me, that most of those rules trigger twice even with runmode single. Can anyone confirm and has an idea why?

sudo /usr/bin/suricata -c /etc/suricata/suricata.yaml -S /tmp/byteextract.rules -r /tmp/byte_extract_test.pcap --runmode=single
sudo tail -f /var/log/suricata/eve.json| jq 'select(.event_type=="alert")'

{
  "timestamp": "2019-06-14T17:45:12.603582+0200",
  "flow_id": 1614610294124411,
  "pcap_cnt": 4,
  "event_type": "alert",
  "src_ip": "10.0.0.76",
  "src_port": 5362,
  "dest_ip": "10.0.0.199",
  "dest_port": 80,
  "proto": "TCP",
  "alert": {
    "action": "allowed",
    "gid": 1,
    "signature_id": 1,
    "rev": 1,
    "signature": "byte extract test 1",
    "category": "",
    "severity": 3
  },
  "flow": {
    "pkts_toserver": 3,
    "pkts_toclient": 1,
    "bytes_toserver": 200,
    "bytes_toclient": 66,
    "start": "2019-06-14T17:45:12.601979+0200" 
  }
}
{
  "timestamp": "2019-06-14T17:45:12.610278+0200",
  "flow_id": 1614610294124411,
  "event_type": "alert",
  "src_ip": "10.0.0.76",
  "src_port": 5362,
  "dest_ip": "10.0.0.199",
  "dest_port": 80,
  "proto": "TCP",
  "alert": {
    "action": "allowed",
    "gid": 1,
    "signature_id": 1,
    "rev": 1,
    "signature": "byte extract test 1",
    "category": "",
    "severity": 3
  },
  "http": {
    "http_port": 0,
    "url": "!34.\u0011\"3DUf.21",
    "http_content_type": "text/html",
    "http_method": "1026\u00108",
    "status": 400,
    "length": 182
  },
  "app_proto": "http",
  "app_proto_ts": "failed",
  "flow": {
    "pkts_toserver": 5,
    "pkts_toclient": 5,
    "bytes_toserver": 320,
    "bytes_toclient": 643,
    "start": "2019-06-14T17:45:12.601979+0200" 
  }
}

The second timestamp would match the last ACK packet but that doesn't have the content.
I can also reproduce it without byte_extract with this rule:
alert tcp any any -> any any (msg:"byte extract test 1"; content:"|33 34|"; offset:0; depth:10; sid:1; rev:1;)

Actions #5

Updated by Victor Julien over 2 years ago

This is probably because the rule matches on both packet payload and stream payload. --engine-analysis could be used to confirm.

Wrt the pcap/rules. Yeah you can use them in the tests. Just add a readme with a link to the ticket.

Actions #6

Updated by Andreas Herz over 2 years ago

  • Tracker changed from Support to Bug
  • Status changed from New to Assigned
  • Target version changed from Support to 70
Actions #7

Updated by Victor Julien about 2 years ago

  • Status changed from Assigned to New
  • Assignee changed from Andreas Herz to OISF Dev
  • Priority changed from Normal to High
  • Target version changed from 70 to 5.0rc1
Actions #8

Updated by Victor Julien about 2 years ago

  • Target version changed from 5.0rc1 to 70
Actions #10

Updated by Jungho Yoon almost 2 years ago

In my environment it's still the same. Can you tell me the settings for the above test?

New installation for pcap reading.

suricata.yaml is the same as the default value.

[root@readids ~]# suricata --build-info
This is Suricata version 5.0.0 RELEASE
Features: PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_LIBJANSSON PROFILING TLS MAGIC RUST
SIMD support: none
Atomic intrinsics: 1 2 4 8 byte(s)
64-bits, Little-endian architecture
GCC version 9.2.1 20190827 (Red Hat 9.2.1-1), C version 199901
compiled with _FORTIFY_SOURCE=0
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.31, linked against LibHTP v0.5.31

Suricata Configuration:
  AF_PACKET support:                       yes
  eBPF support:                            no
  XDP support:                             no
  PF_RING support:                         no
  NFQueue support:                         no
  NFLOG support:                           no
  IPFW support:                            no
  Netmap support:                          no
  DAG enabled:                             no
  Napatech enabled:                        no
  WinDivert enabled:                       no

  Unix socket enabled:                     yes
  Detection enabled:                       yes

  Libmagic support:                        yes
  libnss support:                          yes
  libnspr support:                         yes
  libjansson support:                      yes
  hiredis support:                         no
  hiredis async with libevent:             no
  Prelude support:                         no
  PCRE jit:                                yes
  LUA support:                             no
  libluajit:                               no
  GeoIP2 support:                          no
  Non-bundled htp:                         no
  Old barnyard2 support:                   no
  Hyperscan support:                       yes
  Libnet support:                          yes
  liblz4 support:                          yes

  Rust support:                            yes
  Rust strict mode:                        no
  Rust compiler path:                      /usr/bin/rustc
  Rust compiler version:                   rustc 1.38.0
  Cargo path:                              /usr/bin/cargo
  Cargo version:                           cargo 1.38.0

  Python support:                          yes
  Python path:                             /usr/bin/python3
  Python distutils                         yes
  Python yaml                              yes
  Install suricatactl:                     yes
  Install suricatasc:                      yes
  Install suricata-update:                 yes

  Profiling enabled:                       yes
  Profiling locks enabled:                 no

Development settings:
  Coccinelle / spatch:                     no
  Unit tests enabled:                      no
  Debug output enabled:                    no
  Debug validation enabled:                no

Generic build parameters:
  Installation prefix:                     /usr
  Configuration directory:                 /etc/suricata/
  Log directory:                           /var/log/suricata/

  --prefix                                 /usr
  --sysconfdir                             /etc
  --localstatedir                          /var
  --datarootdir                            /usr/share

  Host:                                    x86_64-pc-linux-gnu
  Compiler:                                gcc (exec name) / gcc (real)
  GCC Protect enabled:                     no
  GCC march native enabled:                no
  GCC Profile enabled:                     no
  Position Independent Executable enabled: no
  CFLAGS                                   -g -O2 -I${srcdir}/../rust/gen/c-headers

[root@readids ~]# suricata -c /etc/suricata/suricata.yaml -k none -r /root/pcap/byte_extract_test.pcap ; cat fast.log
22/10/2019 -- 20:39:00 - <Notice> - This is Suricata version 5.0.0 RELEASE running in USER mode
22/10/2019 -- 20:39:00 - <Warning> - [ERRCODE: SC_WARN_NO_JA3_SUPPORT(308)] - no MD5 calculation support built in (LibNSS), disabling JA3
22/10/2019 -- 20:39:00 - <Notice> - all 5 packet processing threads, 4 management threads initialized, engine started.
22/10/2019 -- 20:39:00 - <Notice> - Signal Received.  Stopping engine.
22/10/2019 -- 20:39:00 - <Notice> - Pcap-file module read 1 files, 10 packets, 963 bytes
06/15/2019-00:45:12.603582  [**] [1:1:1] byte extract test 1 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.76:5362 -> 10.0.0.199:80
06/15/2019-00:45:12.603582  [**] [1:8:1] byte extract test 8 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.76:5362 -> 10.0.0.199:80
06/15/2019-00:45:12.610278  [**] [1:1:1] byte extract test 1 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.76:5362 -> 10.0.0.199:80
06/15/2019-00:45:12.610278  [**] [1:4:1] byte extract test 4 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.76:5362 -> 10.0.0.199:80
06/15/2019-00:45:12.610278  [**] [1:5:1] byte extract test 5 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.76:5362 -> 10.0.0.199:80
06/15/2019-00:45:12.610278  [**] [1:8:1] byte extract test 8 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.76:5362 -> 10.0.0.199:80
[root@readids ~]# cat suricata.log
22/10/2019 -- 20:39:00 - <Notice> - This is Suricata version 5.0.0 RELEASE running in USER mode
22/10/2019 -- 20:39:00 - <Info> - CPUs/cores online: 4
22/10/2019 -- 20:39:00 - <Warning> - [ERRCODE: SC_WARN_NO_JA3_SUPPORT(308)] - no MD5 calculation support built in (LibNSS), disabling JA3
22/10/2019 -- 20:39:00 - <Info> - fast output device (regular) initialized: fast.log
22/10/2019 -- 20:39:00 - <Info> - eve-log output device (regular) initialized: eve.json
22/10/2019 -- 20:39:00 - <Info> - stats output device (regular) initialized: stats.log
22/10/2019 -- 20:39:00 - <Info> - 1 rule files processed. 10 rules successfully loaded, 0 rules failed
22/10/2019 -- 20:39:00 - <Info> - Threshold config parsed: 0 rule(s) found
22/10/2019 -- 20:39:00 - <Info> - 10 signatures processed. 0 are IP-only rules, 10 are inspecting packet payload, 0 inspect application layer, 0 are decoder event only
22/10/2019 -- 20:39:00 - <Notice> - all 5 packet processing threads, 4 management threads initialized, engine started.
22/10/2019 -- 20:39:00 - <Info> - Starting file run for /root/pcap/byte_extract_test.pcap
22/10/2019 -- 20:39:00 - <Info> - pcap file /root/pcap/byte_extract_test.pcap end of file reached (pcap err code 0)
22/10/2019 -- 20:39:00 - <Notice> - Signal Received.  Stopping engine.
22/10/2019 -- 20:39:00 - <Info> - time elapsed 0.031s
22/10/2019 -- 20:39:00 - <Notice> - Pcap-file module read 1 files, 10 packets, 963 bytes
22/10/2019 -- 20:39:00 - <Info> - Alerts: 6
22/10/2019 -- 20:39:00 - <Info> - cleaning up signature grouping structure... complete

Actions #11

Updated by Victor Julien almost 2 years ago

  • Status changed from New to Assigned
  • Assignee changed from OISF Dev to Jeff Lucovsky

Jeff, since you are also looking at byte_math, it might make sense to look at this one at the same time.

Actions #12

Updated by Jeff Lucovsky almost 2 years ago

  • Status changed from Assigned to Closed
Actions #13

Updated by Victor Julien almost 2 years ago

  • Priority changed from High to Normal
  • Target version changed from 70 to 5.0.1
  • Label Needs backport added
Actions #14

Updated by Victor Julien almost 2 years ago

  • Copied to Bug #3369: byte_extract does not work in some situations (4.1.x) added
Actions

Also available in: Atom PDF