Project

General

Profile

Actions

Bug #6567

open

anomaly and file info logs discrepancy results between versions

Added by Peter Manev almost 1 year ago. Updated 12 months ago.

Status:
New
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

Reading the same pcap (attached, thanks to AnyRun) with Suricata 7.0.2 and latest gitmaster gives different results.
Mainly 1 extra fileinfo log with latest master vs Suricata 7.0.2
and 1 extra anomaly log with 7.0.2 vs latest master:

sudo /opt/suritest-profiling/bin/suricata  -S "rules/*.rules"  -l logs/ -k none -r ce7ca983-9e4b-4251-a7c3-fefa3da02ebe.pcap ;  echo "Suricata event types:" ; jq -r .event_type logs/eve.json | sort | uniq -c |sort -rn ; echo "Alerts:" ; grep '"event_type":"alert"' logs/eve.json | jq .alert.signature | sort -rn | uniq -c | sort -rn ; 

Notice: suricata: This is Suricata version 8.0.0-dev (d2b25af3f 2023-11-17) running in USER mode [LogVersion:suricata.c:1148]
Warning: app-layer-htp: Flash decompression is deprecated and will be removed in Suricata 8; see ticket #6179 [HTPConfigParseParameters:app-layer-htp.c:2908]
Notice: threads: Threads created -> RX: 1 W: 16 FM: 1 FR: 1   Engine started. [TmThreadWaitOnThreadRunning:tm-threads.c:1893]
Notice: suricata: Signal Received.  Stopping engine. [SuricataMainLoop:suricata.c:2807]
Notice: pcap: read 1 file, 5593 packets, 3650074 bytes [ReceivePcapFileThreadExitStats:source-pcap-file.c:388]
Suricata event types:
     18 flow
      8 fileinfo
      7 alert
      4 http
      2 dns
      1 tls
      1 stats
Alerts:
      1 "ETPRO POLICY Observed External IP Lookup Domain (api.ip .sb in TLS SNI)" 
      1 "ETPRO POLICY External IP Address Lookup DNS Query (api .ip .sb)" 
      1 "ETPRO MALWARE RedLine - SetEnvironment Request" 
      1 "ETPRO MALWARE RedLine - EnvironmentSettings Request" 
      1 "ETPRO MALWARE RedLine - CheckConnect Request" 
      1 "ET ATTACK_RESPONSE Win32/LeftHook Stealer Browser Extension Config Inbound" 
      1 "ET ATTACK_RESPONSE RedLine Stealer - CheckConnect Response" 

sudo /opt/suritest702/bin/suricata  -S "rules/*.rules"  -l logs/ -k none -r ce7ca983-9e4b-4251-a7c3-fefa3da02ebe.pcap ;  echo "Suricata event types:" ; jq -r .event_type logs/eve.json | sort | uniq -c |sort -rn ; echo "Alerts:" ; grep '"event_type":"alert"' logs/eve.json | jq .alert.signature | sort -rn | uniq -c | sort -rn ; 

i: suricata: This is Suricata version 7.0.2 RELEASE running in USER mode
i: threads: Threads created -> RX: 1 W: 16 FM: 1 FR: 1   Engine started.
i: suricata: Signal Received.  Stopping engine.
i: pcap: read 1 file, 5593 packets, 3650074 bytes
Suricata event types:
     18 flow
      7 fileinfo
      7 alert
      4 http
      2 dns
      1 tls
      1 stats
      1 anomaly
Alerts:
      1 "ETPRO POLICY Observed External IP Lookup Domain (api.ip .sb in TLS SNI)" 
      1 "ETPRO POLICY External IP Address Lookup DNS Query (api .ip .sb)" 
      1 "ETPRO MALWARE RedLine - SetEnvironment Request" 
      1 "ETPRO MALWARE RedLine - EnvironmentSettings Request" 
      1 "ETPRO MALWARE RedLine - CheckConnect Request" 
      1 "ET ATTACK_RESPONSE Win32/LeftHook Stealer Browser Extension Config Inbound" 
      1 "ET ATTACK_RESPONSE RedLine Stealer - CheckConnect Response" 

grep '"event_type":"anomaly"' logs/eve.json | jq . 
{
  "timestamp": "2023-09-18T08:13:58.882971+0200",
  "flow_id": 1676748533473995,
  "pcap_cnt": 2878,
  "event_type": "anomaly",
  "src_ip": "192.168.100.237",
  "src_port": 49175,
  "dest_ip": "142.11.240.191",
  "dest_port": 35361,
  "proto": "TCP",
  "pkt_src": "wire/pcap",
  "metadata": {
    "flowbits": [
      "http.dottedquadhost" 
    ]
  },
  "tx_id": 3,
  "anomaly": {
    "app_proto": "http",
    "type": "applayer",
    "event": "UNABLE_TO_MATCH_RESPONSE_TO_REQUEST",
    "layer": "proto_parser" 
  }
}

Files

Actions #2

Updated by Peter Manev almost 1 year ago

password is "infected"

Actions #3

Updated by Jason Ish almost 1 year ago

I'm unable to replicate. I used git master and suricata-7.0.2 and I get the same output for both:

Suricata event types:
     18 flow
      7 fileinfo
      4 http
      2 dns
      1 tls
      1 stats
      1 anomaly
      1 alert
Alerts:
      1 "SURICATA HTTP unable to match response to request" 

My command was very similar to yours, only paths changed:

./src/suricata  -S "rules/*.rules"  -l . -k none -r ~/scratch/ce7ca983-9e4b-4251-a7c3-fefa3da02ebe.pcap ;  echo "Suricata event types:" ; jq -r .event_type ./eve.json | sort | uniq -c |sort -rn ; echo "Alerts:" ; grep '"event_type":"alert"' ./eve.json | jq .alert.signature | sort -rn | uniq -c | sort -rn ; 

I have not looked into the pcap to see what should be correct.

Actions #4

Updated by Jason Ish almost 1 year ago

I don't have et/pro, but with et/open, still the same output for both versions:

Suricata event types:
     18 flow
      7 fileinfo
      4 http
      3 alert
      2 dns
      1 tls
      1 stats
      1 anomaly
Alerts:
      1 "SURICATA HTTP unable to match response to request" 
      1 "ET ATTACK_RESPONSE Win32/LeftHook Stealer Browser Extension Config Inbound" 
      1 "ET ATTACK_RESPONSE RedLine Stealer - CheckConnect Response" 

Actions #5

Updated by Peter Manev almost 1 year ago

What is the build info for your master build ?

Actions #6

Updated by Jason Ish almost 1 year ago

Build info for master:

This is Suricata version 8.0.0-dev (c272a646c5 2023-11-21)
Features: DEBUG UNITTESTS NFQ PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HTTP2_DECOMPRESSION HAVE_LUA HAVE_LIBJANSSON TLS TLS_C11 MAGIC RUST ASAN 
SIMD support: SSE_4_2 SSE_4_1 SSE_3 
Atomic intrinsics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 13.2.1 20231011 (Red Hat 13.2.1-4), C version 201112
compiled with _FORTIFY_SOURCE=0
L1 cache line size (CLS)=64
thread local storage method: _Thread_local
compiled with LibHTP v0.5.45, linked against LibHTP v0.5.45

Suricata Configuration:
  AF_PACKET support:                       yes
  AF_XDP support:                          no
  DPDK support:                            no
  eBPF support:                            no
  XDP support:                             no
  PF_RING support:                         no
  NFQueue support:                         yes
  NFLOG support:                           no
  IPFW support:                            no
  Netmap support:                          no 
  DAG enabled:                             no
  Napatech enabled:                        no
  WinDivert enabled:                       no

  Unix socket enabled:                     yes
  Detection enabled:                       yes

  Libmagic support:                        yes
  libjansson support:                      yes
  hiredis support:                         no
  hiredis async with libevent:             no
  PCRE jit:                                yes
  LUA support:                             yes
  libluajit:                               no
  GeoIP2 support:                          no
  Non-bundled htp:                         no
  Hyperscan support:                       yes
  Libnet support:                          yes
  liblz4 support:                          yes
  Landlock support:                        yes

  Rust support:                            yes
  Rust strict mode:                        no
  Rust compiler path:                      /home/jason/.cargo/bin/rustc
  Rust compiler version:                   rustc 1.73.0 (cc66ad468 2023-10-03)
  Cargo path:                              /home/jason/.cargo/bin/cargo
  Cargo version:                           cargo 1.73.0 (9c4383fb5 2023-08-26)

  Python support:                          yes
  Python path:                             /usr/bin/python3
  Install suricatactl:                     yes
  Install suricatasc:                      yes
  Install suricata-update:                 yes

  Profiling enabled:                       no
  Profiling locks enabled:                 no
  Profiling rules enabled:                 no

  Plugin support (experimental):           yes
  DPDK Bond PMD:                           no

Development settings:
  Coccinelle / spatch:                     no
  Unit tests enabled:                      yes
  Debug output enabled:                    yes
  Debug validation enabled:                no
  Fuzz targets enabled:                    no

Generic build parameters:
  Installation prefix:                     /opt/suricata/8.0.0-dev
  Configuration directory:                 /opt/suricata/8.0.0-dev/etc/suricata/
  Log directory:                           /opt/suricata/8.0.0-dev/var/log/suricata/

  --prefix                                 /opt/suricata/8.0.0-dev
  --sysconfdir                             /opt/suricata/8.0.0-dev/etc
  --localstatedir                          /opt/suricata/8.0.0-dev/var
  --datarootdir                            /opt/suricata/8.0.0-dev/share

  Host:                                    x86_64-pc-linux-gnu
  Compiler:                                gcc (exec name) / g++ (real)
  GCC Protect enabled:                     no
  GCC march native enabled:                yes
  GCC Profile enabled:                     no
  Position Independent Executable enabled: no
  CFLAGS                                   -fno-common -Wall -Wextra -Wchar-subscripts -Wno-unused-parameter -Wno-unused-function -Wno-deprecated-declarations -ggdb3 -O0 -fsanitize=address -fno-omit-frame-pointer -fno-inline -fPIC -std=c11 -march=native -I${srcdir}/../rust/gen -I${srcdir}/../rust/dist
  PCAP_CFLAGS                               -I/usr/include
  SECCFLAGS                                

Actions #7

Updated by Peter Manev 12 months ago

It took me a bit to pinpoint the problem but this can be triggered the following way - by adjusting the stream.depth value.
Using latest master when i adjust the value to 2mb , I loose one anomaly event and gain one fileinfo event:

rm logs/* -rf ;  suricata  -S /dev/null  -l logs/ -k none -r ce7ca983-9e4b-4251-a7c3-fefa3da02ebe.pcap ;  echo "Suricata event types:" ; jq -r .event_type logs/eve.json | sort | uniq -c |sort -rn ; grep '"event_type":"alert"' logs/eve.json | jq .alert.signature | sort -rn | uniq -c | sort -rn ; 

Notice: suricata: This is Suricata version 8.0.0-dev (13cc49388 2023-12-01) running in USER mode [LogVersion:suricata.c:1146]
Notice: threads: Threads created -> RX: 1 W: 16 FM: 1 FR: 1   Engine started. [TmThreadWaitOnThreadRunning:tm-threads.c:1891]
Notice: suricata: Signal Received.  Stopping engine. [SuricataMainLoop:suricata.c:2805]
Notice: pcap: read 1 file, 5593 packets, 3650074 bytes [ReceivePcapFileThreadExitStats:source-pcap-file.c:387]
Suricata event types:
     18 flow
      7 fileinfo
      4 http
      2 dns
      1 tls
      1 stats
      1 anomaly

rm logs/* -rf ; sudo suricata  -S /dev/null  -l logs/ -k none -r ce7ca983-9e4b-4251-a7c3-fefa3da02ebe.pcap --set "stream.reassembly.depth=2mb" ;  echo "Suricata event types:" ; jq -r .event_type logs/eve.json | sort | uniq -c |sort -rn ; grep '"event_type":"alert"' logs/eve.json | jq .alert.signature | sort -rn | uniq -c | sort -rn ; 

Notice: suricata: This is Suricata version 8.0.0-dev (13cc49388 2023-12-01) running in USER mode [LogVersion:suricata.c:1146]
Notice: threads: Threads created -> RX: 1 W: 16 FM: 1 FR: 1   Engine started. [TmThreadWaitOnThreadRunning:tm-threads.c:1891]
Notice: suricata: Signal Received.  Stopping engine. [SuricataMainLoop:suricata.c:2805]
Notice: pcap: read 1 file, 5593 packets, 3650074 bytes [ReceivePcapFileThreadExitStats:source-pcap-file.c:387]
Suricata event types:
     18 flow
      8 fileinfo
      4 http
      2 dns
      1 tls
      1 stats

This is the anomaly event:

{
  "timestamp": "2023-09-18T08:13:58.882971+0200",
  "flow_id": 1676750216978051,
  "pcap_cnt": 2878,
  "event_type": "anomaly",
  "src_ip": "192.168.100.237",
  "src_port": 49175,
  "dest_ip": "142.11.240.191",
  "dest_port": 35361,
  "proto": "TCP",
  "pkt_src": "wire/pcap",
  "tx_id": 3,
  "anomaly": {
    "app_proto": "http",
    "type": "applayer",
    "event": "UNABLE_TO_MATCH_RESPONSE_TO_REQUEST",
    "layer": "proto_parser" 
  }
}
Actions

Also available in: Atom PDF