Suricata

sure, can we use those rules and this pcap or do we need dedicated permission?

I also confirmed that as soon as I replace the vars with the value it works as expected, so no issue with the keywords like distance, depth IMHO.

What's also confusing to me, that most of those rules trigger twice even with runmode single. Can anyone confirm and has an idea why?

sudo /usr/bin/suricata -c /etc/suricata/suricata.yaml -S /tmp/byteextract.rules -r /tmp/byte_extract_test.pcap --runmode=single
sudo tail -f /var/log/suricata/eve.json| jq 'select(.event_type=="alert")'

{
  "timestamp": "2019-06-14T17:45:12.603582+0200",
  "flow_id": 1614610294124411,
  "pcap_cnt": 4,
  "event_type": "alert",
  "src_ip": "10.0.0.76",
  "src_port": 5362,
  "dest_ip": "10.0.0.199",
  "dest_port": 80,
  "proto": "TCP",
  "alert": {
    "action": "allowed",
    "gid": 1,
    "signature_id": 1,
    "rev": 1,
    "signature": "byte extract test 1",
    "category": "",
    "severity": 3
  },
  "flow": {
    "pkts_toserver": 3,
    "pkts_toclient": 1,
    "bytes_toserver": 200,
    "bytes_toclient": 66,
    "start": "2019-06-14T17:45:12.601979+0200" 
  }
}
{
  "timestamp": "2019-06-14T17:45:12.610278+0200",
  "flow_id": 1614610294124411,
  "event_type": "alert",
  "src_ip": "10.0.0.76",
  "src_port": 5362,
  "dest_ip": "10.0.0.199",
  "dest_port": 80,
  "proto": "TCP",
  "alert": {
    "action": "allowed",
    "gid": 1,
    "signature_id": 1,
    "rev": 1,
    "signature": "byte extract test 1",
    "category": "",
    "severity": 3
  },
  "http": {
    "http_port": 0,
    "url": "!34.\u0011\"3DUf.21",
    "http_content_type": "text/html",
    "http_method": "1026\u00108",
    "status": 400,
    "length": 182
  },
  "app_proto": "http",
  "app_proto_ts": "failed",
  "flow": {
    "pkts_toserver": 5,
    "pkts_toclient": 5,
    "bytes_toserver": 320,
    "bytes_toclient": 643,
    "start": "2019-06-14T17:45:12.601979+0200" 
  }
}

alert tcp any any -> any any (msg:"byte extract test 1"; content:"|33 34|"; offset:0; depth:10; sid:1; rev:1;)

This is probably because the rule matches on both packet payload and stream payload. --engine-analysis could be used to confirm.

Wrt the pcap/rules. Yeah you can use them in the tests. Just add a readme with a link to the ticket.

Test cases: https://github.com/OISF/suricata-verify/pull/86

In my environment it's still the same. Can you tell me the settings for the above test?

New installation for pcap reading.

suricata.yaml is the same as the default value.

[root@readids ~]# suricata --build-info
This is Suricata version 5.0.0 RELEASE
Features: PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_LIBJANSSON PROFILING TLS MAGIC RUST
SIMD support: none
Atomic intrinsics: 1 2 4 8 byte(s)
64-bits, Little-endian architecture
GCC version 9.2.1 20190827 (Red Hat 9.2.1-1), C version 199901
compiled with _FORTIFY_SOURCE=0
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.31, linked against LibHTP v0.5.31

Suricata Configuration:
  AF_PACKET support:                       yes
  eBPF support:                            no
  XDP support:                             no
  PF_RING support:                         no
  NFQueue support:                         no
  NFLOG support:                           no
  IPFW support:                            no
  Netmap support:                          no
  DAG enabled:                             no
  Napatech enabled:                        no
  WinDivert enabled:                       no

  Unix socket enabled:                     yes
  Detection enabled:                       yes

  Libmagic support:                        yes
  libnss support:                          yes
  libnspr support:                         yes
  libjansson support:                      yes
  hiredis support:                         no
  hiredis async with libevent:             no
  Prelude support:                         no
  PCRE jit:                                yes
  LUA support:                             no
  libluajit:                               no
  GeoIP2 support:                          no
  Non-bundled htp:                         no
  Old barnyard2 support:                   no
  Hyperscan support:                       yes
  Libnet support:                          yes
  liblz4 support:                          yes

  Rust support:                            yes
  Rust strict mode:                        no
  Rust compiler path:                      /usr/bin/rustc
  Rust compiler version:                   rustc 1.38.0
  Cargo path:                              /usr/bin/cargo
  Cargo version:                           cargo 1.38.0

  Python support:                          yes
  Python path:                             /usr/bin/python3
  Python distutils                         yes
  Python yaml                              yes
  Install suricatactl:                     yes
  Install suricatasc:                      yes
  Install suricata-update:                 yes

  Profiling enabled:                       yes
  Profiling locks enabled:                 no

Development settings:
  Coccinelle / spatch:                     no
  Unit tests enabled:                      no
  Debug output enabled:                    no
  Debug validation enabled:                no

Generic build parameters:
  Installation prefix:                     /usr
  Configuration directory:                 /etc/suricata/
  Log directory:                           /var/log/suricata/

  --prefix                                 /usr
  --sysconfdir                             /etc
  --localstatedir                          /var
  --datarootdir                            /usr/share

  Host:                                    x86_64-pc-linux-gnu
  Compiler:                                gcc (exec name) / gcc (real)
  GCC Protect enabled:                     no
  GCC march native enabled:                no
  GCC Profile enabled:                     no
  Position Independent Executable enabled: no
  CFLAGS                                   -g -O2 -I${srcdir}/../rust/gen/c-headers

[root@readids ~]# suricata -c /etc/suricata/suricata.yaml -k none -r /root/pcap/byte_extract_test.pcap ; cat fast.log
22/10/2019 -- 20:39:00 - <Notice> - This is Suricata version 5.0.0 RELEASE running in USER mode
22/10/2019 -- 20:39:00 - <Warning> - [ERRCODE: SC_WARN_NO_JA3_SUPPORT(308)] - no MD5 calculation support built in (LibNSS), disabling JA3
22/10/2019 -- 20:39:00 - <Notice> - all 5 packet processing threads, 4 management threads initialized, engine started.
22/10/2019 -- 20:39:00 - <Notice> - Signal Received.  Stopping engine.
22/10/2019 -- 20:39:00 - <Notice> - Pcap-file module read 1 files, 10 packets, 963 bytes
06/15/2019-00:45:12.603582  [**] [1:1:1] byte extract test 1 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.76:5362 -> 10.0.0.199:80
06/15/2019-00:45:12.603582  [**] [1:8:1] byte extract test 8 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.76:5362 -> 10.0.0.199:80
06/15/2019-00:45:12.610278  [**] [1:1:1] byte extract test 1 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.76:5362 -> 10.0.0.199:80
06/15/2019-00:45:12.610278  [**] [1:4:1] byte extract test 4 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.76:5362 -> 10.0.0.199:80
06/15/2019-00:45:12.610278  [**] [1:5:1] byte extract test 5 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.76:5362 -> 10.0.0.199:80
06/15/2019-00:45:12.610278  [**] [1:8:1] byte extract test 8 [**] [Classification: (null)] [Priority: 3] {TCP} 10.0.0.76:5362 -> 10.0.0.199:80
[root@readids ~]# cat suricata.log
22/10/2019 -- 20:39:00 - <Notice> - This is Suricata version 5.0.0 RELEASE running in USER mode
22/10/2019 -- 20:39:00 - <Info> - CPUs/cores online: 4
22/10/2019 -- 20:39:00 - <Warning> - [ERRCODE: SC_WARN_NO_JA3_SUPPORT(308)] - no MD5 calculation support built in (LibNSS), disabling JA3
22/10/2019 -- 20:39:00 - <Info> - fast output device (regular) initialized: fast.log
22/10/2019 -- 20:39:00 - <Info> - eve-log output device (regular) initialized: eve.json
22/10/2019 -- 20:39:00 - <Info> - stats output device (regular) initialized: stats.log
22/10/2019 -- 20:39:00 - <Info> - 1 rule files processed. 10 rules successfully loaded, 0 rules failed
22/10/2019 -- 20:39:00 - <Info> - Threshold config parsed: 0 rule(s) found
22/10/2019 -- 20:39:00 - <Info> - 10 signatures processed. 0 are IP-only rules, 10 are inspecting packet payload, 0 inspect application layer, 0 are decoder event only
22/10/2019 -- 20:39:00 - <Notice> - all 5 packet processing threads, 4 management threads initialized, engine started.
22/10/2019 -- 20:39:00 - <Info> - Starting file run for /root/pcap/byte_extract_test.pcap
22/10/2019 -- 20:39:00 - <Info> - pcap file /root/pcap/byte_extract_test.pcap end of file reached (pcap err code 0)
22/10/2019 -- 20:39:00 - <Notice> - Signal Received.  Stopping engine.
22/10/2019 -- 20:39:00 - <Info> - time elapsed 0.031s
22/10/2019 -- 20:39:00 - <Notice> - Pcap-file module read 1 files, 10 packets, 963 bytes
22/10/2019 -- 20:39:00 - <Info> - Alerts: 6
22/10/2019 -- 20:39:00 - <Info> - cleaning up signature grouping structure... complete