Bug #6567
openanomaly and file info logs discrepancy results between versions
Description
Reading the same pcap (attached, thanks to AnyRun) with Suricata 7.0.2 and latest gitmaster gives different results.
Mainly 1 extra fileinfo log with latest master vs Suricata 7.0.2
and 1 extra anomaly log with 7.0.2 vs latest master:
sudo /opt/suritest-profiling/bin/suricata -S "rules/*.rules" -l logs/ -k none -r ce7ca983-9e4b-4251-a7c3-fefa3da02ebe.pcap ; echo "Suricata event types:" ; jq -r .event_type logs/eve.json | sort | uniq -c |sort -rn ; echo "Alerts:" ; grep '"event_type":"alert"' logs/eve.json | jq .alert.signature | sort -rn | uniq -c | sort -rn ; Notice: suricata: This is Suricata version 8.0.0-dev (d2b25af3f 2023-11-17) running in USER mode [LogVersion:suricata.c:1148] Warning: app-layer-htp: Flash decompression is deprecated and will be removed in Suricata 8; see ticket #6179 [HTPConfigParseParameters:app-layer-htp.c:2908] Notice: threads: Threads created -> RX: 1 W: 16 FM: 1 FR: 1 Engine started. [TmThreadWaitOnThreadRunning:tm-threads.c:1893] Notice: suricata: Signal Received. Stopping engine. [SuricataMainLoop:suricata.c:2807] Notice: pcap: read 1 file, 5593 packets, 3650074 bytes [ReceivePcapFileThreadExitStats:source-pcap-file.c:388] Suricata event types: 18 flow 8 fileinfo 7 alert 4 http 2 dns 1 tls 1 stats Alerts: 1 "ETPRO POLICY Observed External IP Lookup Domain (api.ip .sb in TLS SNI)" 1 "ETPRO POLICY External IP Address Lookup DNS Query (api .ip .sb)" 1 "ETPRO MALWARE RedLine - SetEnvironment Request" 1 "ETPRO MALWARE RedLine - EnvironmentSettings Request" 1 "ETPRO MALWARE RedLine - CheckConnect Request" 1 "ET ATTACK_RESPONSE Win32/LeftHook Stealer Browser Extension Config Inbound" 1 "ET ATTACK_RESPONSE RedLine Stealer - CheckConnect Response" sudo /opt/suritest702/bin/suricata -S "rules/*.rules" -l logs/ -k none -r ce7ca983-9e4b-4251-a7c3-fefa3da02ebe.pcap ; echo "Suricata event types:" ; jq -r .event_type logs/eve.json | sort | uniq -c |sort -rn ; echo "Alerts:" ; grep '"event_type":"alert"' logs/eve.json | jq .alert.signature | sort -rn | uniq -c | sort -rn ; i: suricata: This is Suricata version 7.0.2 RELEASE running in USER mode i: threads: Threads created -> RX: 1 W: 16 FM: 1 FR: 1 Engine started. i: suricata: Signal Received. Stopping engine. i: pcap: read 1 file, 5593 packets, 3650074 bytes Suricata event types: 18 flow 7 fileinfo 7 alert 4 http 2 dns 1 tls 1 stats 1 anomaly Alerts: 1 "ETPRO POLICY Observed External IP Lookup Domain (api.ip .sb in TLS SNI)" 1 "ETPRO POLICY External IP Address Lookup DNS Query (api .ip .sb)" 1 "ETPRO MALWARE RedLine - SetEnvironment Request" 1 "ETPRO MALWARE RedLine - EnvironmentSettings Request" 1 "ETPRO MALWARE RedLine - CheckConnect Request" 1 "ET ATTACK_RESPONSE Win32/LeftHook Stealer Browser Extension Config Inbound" 1 "ET ATTACK_RESPONSE RedLine Stealer - CheckConnect Response" grep '"event_type":"anomaly"' logs/eve.json | jq . { "timestamp": "2023-09-18T08:13:58.882971+0200", "flow_id": 1676748533473995, "pcap_cnt": 2878, "event_type": "anomaly", "src_ip": "192.168.100.237", "src_port": 49175, "dest_ip": "142.11.240.191", "dest_port": 35361, "proto": "TCP", "pkt_src": "wire/pcap", "metadata": { "flowbits": [ "http.dottedquadhost" ] }, "tx_id": 3, "anomaly": { "app_proto": "http", "type": "applayer", "event": "UNABLE_TO_MATCH_RESPONSE_TO_REQUEST", "layer": "proto_parser" } }
Files
Updated by Peter Manev about 1 year ago
Updated by Jason Ish about 1 year ago
I'm unable to replicate. I used git master and suricata-7.0.2
and I get the same output for both:
Suricata event types: 18 flow 7 fileinfo 4 http 2 dns 1 tls 1 stats 1 anomaly 1 alert Alerts: 1 "SURICATA HTTP unable to match response to request"
My command was very similar to yours, only paths changed:
./src/suricata -S "rules/*.rules" -l . -k none -r ~/scratch/ce7ca983-9e4b-4251-a7c3-fefa3da02ebe.pcap ; echo "Suricata event types:" ; jq -r .event_type ./eve.json | sort | uniq -c |sort -rn ; echo "Alerts:" ; grep '"event_type":"alert"' ./eve.json | jq .alert.signature | sort -rn | uniq -c | sort -rn ;
I have not looked into the pcap to see what should be correct.
Updated by Jason Ish about 1 year ago
I don't have et/pro, but with et/open, still the same output for both versions:
Suricata event types: 18 flow 7 fileinfo 4 http 3 alert 2 dns 1 tls 1 stats 1 anomaly Alerts: 1 "SURICATA HTTP unable to match response to request" 1 "ET ATTACK_RESPONSE Win32/LeftHook Stealer Browser Extension Config Inbound" 1 "ET ATTACK_RESPONSE RedLine Stealer - CheckConnect Response"
Updated by Peter Manev about 1 year ago
What is the build info for your master build ?
Updated by Jason Ish about 1 year ago
Build info for master:
This is Suricata version 8.0.0-dev (c272a646c5 2023-11-21) Features: DEBUG UNITTESTS NFQ PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HTTP2_DECOMPRESSION HAVE_LUA HAVE_LIBJANSSON TLS TLS_C11 MAGIC RUST ASAN SIMD support: SSE_4_2 SSE_4_1 SSE_3 Atomic intrinsics: 1 2 4 8 16 byte(s) 64-bits, Little-endian architecture GCC version 13.2.1 20231011 (Red Hat 13.2.1-4), C version 201112 compiled with _FORTIFY_SOURCE=0 L1 cache line size (CLS)=64 thread local storage method: _Thread_local compiled with LibHTP v0.5.45, linked against LibHTP v0.5.45 Suricata Configuration: AF_PACKET support: yes AF_XDP support: no DPDK support: no eBPF support: no XDP support: no PF_RING support: no NFQueue support: yes NFLOG support: no IPFW support: no Netmap support: no DAG enabled: no Napatech enabled: no WinDivert enabled: no Unix socket enabled: yes Detection enabled: yes Libmagic support: yes libjansson support: yes hiredis support: no hiredis async with libevent: no PCRE jit: yes LUA support: yes libluajit: no GeoIP2 support: no Non-bundled htp: no Hyperscan support: yes Libnet support: yes liblz4 support: yes Landlock support: yes Rust support: yes Rust strict mode: no Rust compiler path: /home/jason/.cargo/bin/rustc Rust compiler version: rustc 1.73.0 (cc66ad468 2023-10-03) Cargo path: /home/jason/.cargo/bin/cargo Cargo version: cargo 1.73.0 (9c4383fb5 2023-08-26) Python support: yes Python path: /usr/bin/python3 Install suricatactl: yes Install suricatasc: yes Install suricata-update: yes Profiling enabled: no Profiling locks enabled: no Profiling rules enabled: no Plugin support (experimental): yes DPDK Bond PMD: no Development settings: Coccinelle / spatch: no Unit tests enabled: yes Debug output enabled: yes Debug validation enabled: no Fuzz targets enabled: no Generic build parameters: Installation prefix: /opt/suricata/8.0.0-dev Configuration directory: /opt/suricata/8.0.0-dev/etc/suricata/ Log directory: /opt/suricata/8.0.0-dev/var/log/suricata/ --prefix /opt/suricata/8.0.0-dev --sysconfdir /opt/suricata/8.0.0-dev/etc --localstatedir /opt/suricata/8.0.0-dev/var --datarootdir /opt/suricata/8.0.0-dev/share Host: x86_64-pc-linux-gnu Compiler: gcc (exec name) / g++ (real) GCC Protect enabled: no GCC march native enabled: yes GCC Profile enabled: no Position Independent Executable enabled: no CFLAGS -fno-common -Wall -Wextra -Wchar-subscripts -Wno-unused-parameter -Wno-unused-function -Wno-deprecated-declarations -ggdb3 -O0 -fsanitize=address -fno-omit-frame-pointer -fno-inline -fPIC -std=c11 -march=native -I${srcdir}/../rust/gen -I${srcdir}/../rust/dist PCAP_CFLAGS -I/usr/include SECCFLAGS
Updated by Peter Manev about 1 year ago
It took me a bit to pinpoint the problem but this can be triggered the following way - by adjusting the stream.depth value.
Using latest master when i adjust the value to 2mb , I loose one anomaly event and gain one fileinfo event:
rm logs/* -rf ; suricata -S /dev/null -l logs/ -k none -r ce7ca983-9e4b-4251-a7c3-fefa3da02ebe.pcap ; echo "Suricata event types:" ; jq -r .event_type logs/eve.json | sort | uniq -c |sort -rn ; grep '"event_type":"alert"' logs/eve.json | jq .alert.signature | sort -rn | uniq -c | sort -rn ; Notice: suricata: This is Suricata version 8.0.0-dev (13cc49388 2023-12-01) running in USER mode [LogVersion:suricata.c:1146] Notice: threads: Threads created -> RX: 1 W: 16 FM: 1 FR: 1 Engine started. [TmThreadWaitOnThreadRunning:tm-threads.c:1891] Notice: suricata: Signal Received. Stopping engine. [SuricataMainLoop:suricata.c:2805] Notice: pcap: read 1 file, 5593 packets, 3650074 bytes [ReceivePcapFileThreadExitStats:source-pcap-file.c:387] Suricata event types: 18 flow 7 fileinfo 4 http 2 dns 1 tls 1 stats 1 anomaly rm logs/* -rf ; sudo suricata -S /dev/null -l logs/ -k none -r ce7ca983-9e4b-4251-a7c3-fefa3da02ebe.pcap --set "stream.reassembly.depth=2mb" ; echo "Suricata event types:" ; jq -r .event_type logs/eve.json | sort | uniq -c |sort -rn ; grep '"event_type":"alert"' logs/eve.json | jq .alert.signature | sort -rn | uniq -c | sort -rn ; Notice: suricata: This is Suricata version 8.0.0-dev (13cc49388 2023-12-01) running in USER mode [LogVersion:suricata.c:1146] Notice: threads: Threads created -> RX: 1 W: 16 FM: 1 FR: 1 Engine started. [TmThreadWaitOnThreadRunning:tm-threads.c:1891] Notice: suricata: Signal Received. Stopping engine. [SuricataMainLoop:suricata.c:2805] Notice: pcap: read 1 file, 5593 packets, 3650074 bytes [ReceivePcapFileThreadExitStats:source-pcap-file.c:387] Suricata event types: 18 flow 8 fileinfo 4 http 2 dns 1 tls 1 stats
This is the anomaly event:
{ "timestamp": "2023-09-18T08:13:58.882971+0200", "flow_id": 1676750216978051, "pcap_cnt": 2878, "event_type": "anomaly", "src_ip": "192.168.100.237", "src_port": 49175, "dest_ip": "142.11.240.191", "dest_port": 35361, "proto": "TCP", "pkt_src": "wire/pcap", "tx_id": 3, "anomaly": { "app_proto": "http", "type": "applayer", "event": "UNABLE_TO_MATCH_RESPONSE_TO_REQUEST", "layer": "proto_parser" } }