Bug #6567
openanomaly and file info logs discrepancy results between versions
Description
Reading the same pcap (attached, thanks to AnyRun) with Suricata 7.0.2 and latest gitmaster gives different results.
Mainly 1 extra fileinfo log with latest master vs Suricata 7.0.2
and 1 extra anomaly log with 7.0.2 vs latest master:
sudo /opt/suritest-profiling/bin/suricata -S "rules/*.rules" -l logs/ -k none -r ce7ca983-9e4b-4251-a7c3-fefa3da02ebe.pcap ; echo "Suricata event types:" ; jq -r .event_type logs/eve.json | sort | uniq -c |sort -rn ; echo "Alerts:" ; grep '"event_type":"alert"' logs/eve.json | jq .alert.signature | sort -rn | uniq -c | sort -rn ;
Notice: suricata: This is Suricata version 8.0.0-dev (d2b25af3f 2023-11-17) running in USER mode [LogVersion:suricata.c:1148]
Warning: app-layer-htp: Flash decompression is deprecated and will be removed in Suricata 8; see ticket #6179 [HTPConfigParseParameters:app-layer-htp.c:2908]
Notice: threads: Threads created -> RX: 1 W: 16 FM: 1 FR: 1 Engine started. [TmThreadWaitOnThreadRunning:tm-threads.c:1893]
Notice: suricata: Signal Received. Stopping engine. [SuricataMainLoop:suricata.c:2807]
Notice: pcap: read 1 file, 5593 packets, 3650074 bytes [ReceivePcapFileThreadExitStats:source-pcap-file.c:388]
Suricata event types:
18 flow
8 fileinfo
7 alert
4 http
2 dns
1 tls
1 stats
Alerts:
1 "ETPRO POLICY Observed External IP Lookup Domain (api.ip .sb in TLS SNI)"
1 "ETPRO POLICY External IP Address Lookup DNS Query (api .ip .sb)"
1 "ETPRO MALWARE RedLine - SetEnvironment Request"
1 "ETPRO MALWARE RedLine - EnvironmentSettings Request"
1 "ETPRO MALWARE RedLine - CheckConnect Request"
1 "ET ATTACK_RESPONSE Win32/LeftHook Stealer Browser Extension Config Inbound"
1 "ET ATTACK_RESPONSE RedLine Stealer - CheckConnect Response"
sudo /opt/suritest702/bin/suricata -S "rules/*.rules" -l logs/ -k none -r ce7ca983-9e4b-4251-a7c3-fefa3da02ebe.pcap ; echo "Suricata event types:" ; jq -r .event_type logs/eve.json | sort | uniq -c |sort -rn ; echo "Alerts:" ; grep '"event_type":"alert"' logs/eve.json | jq .alert.signature | sort -rn | uniq -c | sort -rn ;
i: suricata: This is Suricata version 7.0.2 RELEASE running in USER mode
i: threads: Threads created -> RX: 1 W: 16 FM: 1 FR: 1 Engine started.
i: suricata: Signal Received. Stopping engine.
i: pcap: read 1 file, 5593 packets, 3650074 bytes
Suricata event types:
18 flow
7 fileinfo
7 alert
4 http
2 dns
1 tls
1 stats
1 anomaly
Alerts:
1 "ETPRO POLICY Observed External IP Lookup Domain (api.ip .sb in TLS SNI)"
1 "ETPRO POLICY External IP Address Lookup DNS Query (api .ip .sb)"
1 "ETPRO MALWARE RedLine - SetEnvironment Request"
1 "ETPRO MALWARE RedLine - EnvironmentSettings Request"
1 "ETPRO MALWARE RedLine - CheckConnect Request"
1 "ET ATTACK_RESPONSE Win32/LeftHook Stealer Browser Extension Config Inbound"
1 "ET ATTACK_RESPONSE RedLine Stealer - CheckConnect Response"
grep '"event_type":"anomaly"' logs/eve.json | jq .
{
"timestamp": "2023-09-18T08:13:58.882971+0200",
"flow_id": 1676748533473995,
"pcap_cnt": 2878,
"event_type": "anomaly",
"src_ip": "192.168.100.237",
"src_port": 49175,
"dest_ip": "142.11.240.191",
"dest_port": 35361,
"proto": "TCP",
"pkt_src": "wire/pcap",
"metadata": {
"flowbits": [
"http.dottedquadhost"
]
},
"tx_id": 3,
"anomaly": {
"app_proto": "http",
"type": "applayer",
"event": "UNABLE_TO_MATCH_RESPONSE_TO_REQUEST",
"layer": "proto_parser"
}
}
Files
Updated by Peter Manev almost 2 years ago
Updated by Jason Ish almost 2 years ago
I'm unable to replicate. I used git master and suricata-7.0.2 and I get the same output for both:
Suricata event types:
18 flow
7 fileinfo
4 http
2 dns
1 tls
1 stats
1 anomaly
1 alert
Alerts:
1 "SURICATA HTTP unable to match response to request"
My command was very similar to yours, only paths changed:
./src/suricata -S "rules/*.rules" -l . -k none -r ~/scratch/ce7ca983-9e4b-4251-a7c3-fefa3da02ebe.pcap ; echo "Suricata event types:" ; jq -r .event_type ./eve.json | sort | uniq -c |sort -rn ; echo "Alerts:" ; grep '"event_type":"alert"' ./eve.json | jq .alert.signature | sort -rn | uniq -c | sort -rn ;
I have not looked into the pcap to see what should be correct.
Updated by Jason Ish almost 2 years ago
I don't have et/pro, but with et/open, still the same output for both versions:
Suricata event types:
18 flow
7 fileinfo
4 http
3 alert
2 dns
1 tls
1 stats
1 anomaly
Alerts:
1 "SURICATA HTTP unable to match response to request"
1 "ET ATTACK_RESPONSE Win32/LeftHook Stealer Browser Extension Config Inbound"
1 "ET ATTACK_RESPONSE RedLine Stealer - CheckConnect Response"
Updated by Peter Manev almost 2 years ago
What is the build info for your master build ?
Updated by Jason Ish almost 2 years ago
Build info for master:
This is Suricata version 8.0.0-dev (c272a646c5 2023-11-21)
Features: DEBUG UNITTESTS NFQ PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HTTP2_DECOMPRESSION HAVE_LUA HAVE_LIBJANSSON TLS TLS_C11 MAGIC RUST ASAN
SIMD support: SSE_4_2 SSE_4_1 SSE_3
Atomic intrinsics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 13.2.1 20231011 (Red Hat 13.2.1-4), C version 201112
compiled with _FORTIFY_SOURCE=0
L1 cache line size (CLS)=64
thread local storage method: _Thread_local
compiled with LibHTP v0.5.45, linked against LibHTP v0.5.45
Suricata Configuration:
AF_PACKET support: yes
AF_XDP support: no
DPDK support: no
eBPF support: no
XDP support: no
PF_RING support: no
NFQueue support: yes
NFLOG support: no
IPFW support: no
Netmap support: no
DAG enabled: no
Napatech enabled: no
WinDivert enabled: no
Unix socket enabled: yes
Detection enabled: yes
Libmagic support: yes
libjansson support: yes
hiredis support: no
hiredis async with libevent: no
PCRE jit: yes
LUA support: yes
libluajit: no
GeoIP2 support: no
Non-bundled htp: no
Hyperscan support: yes
Libnet support: yes
liblz4 support: yes
Landlock support: yes
Rust support: yes
Rust strict mode: no
Rust compiler path: /home/jason/.cargo/bin/rustc
Rust compiler version: rustc 1.73.0 (cc66ad468 2023-10-03)
Cargo path: /home/jason/.cargo/bin/cargo
Cargo version: cargo 1.73.0 (9c4383fb5 2023-08-26)
Python support: yes
Python path: /usr/bin/python3
Install suricatactl: yes
Install suricatasc: yes
Install suricata-update: yes
Profiling enabled: no
Profiling locks enabled: no
Profiling rules enabled: no
Plugin support (experimental): yes
DPDK Bond PMD: no
Development settings:
Coccinelle / spatch: no
Unit tests enabled: yes
Debug output enabled: yes
Debug validation enabled: no
Fuzz targets enabled: no
Generic build parameters:
Installation prefix: /opt/suricata/8.0.0-dev
Configuration directory: /opt/suricata/8.0.0-dev/etc/suricata/
Log directory: /opt/suricata/8.0.0-dev/var/log/suricata/
--prefix /opt/suricata/8.0.0-dev
--sysconfdir /opt/suricata/8.0.0-dev/etc
--localstatedir /opt/suricata/8.0.0-dev/var
--datarootdir /opt/suricata/8.0.0-dev/share
Host: x86_64-pc-linux-gnu
Compiler: gcc (exec name) / g++ (real)
GCC Protect enabled: no
GCC march native enabled: yes
GCC Profile enabled: no
Position Independent Executable enabled: no
CFLAGS -fno-common -Wall -Wextra -Wchar-subscripts -Wno-unused-parameter -Wno-unused-function -Wno-deprecated-declarations -ggdb3 -O0 -fsanitize=address -fno-omit-frame-pointer -fno-inline -fPIC -std=c11 -march=native -I${srcdir}/../rust/gen -I${srcdir}/../rust/dist
PCAP_CFLAGS -I/usr/include
SECCFLAGS
Updated by Peter Manev almost 2 years ago
It took me a bit to pinpoint the problem but this can be triggered the following way - by adjusting the stream.depth value.
Using latest master when i adjust the value to 2mb , I loose one anomaly event and gain one fileinfo event:
rm logs/* -rf ; suricata -S /dev/null -l logs/ -k none -r ce7ca983-9e4b-4251-a7c3-fefa3da02ebe.pcap ; echo "Suricata event types:" ; jq -r .event_type logs/eve.json | sort | uniq -c |sort -rn ; grep '"event_type":"alert"' logs/eve.json | jq .alert.signature | sort -rn | uniq -c | sort -rn ;
Notice: suricata: This is Suricata version 8.0.0-dev (13cc49388 2023-12-01) running in USER mode [LogVersion:suricata.c:1146]
Notice: threads: Threads created -> RX: 1 W: 16 FM: 1 FR: 1 Engine started. [TmThreadWaitOnThreadRunning:tm-threads.c:1891]
Notice: suricata: Signal Received. Stopping engine. [SuricataMainLoop:suricata.c:2805]
Notice: pcap: read 1 file, 5593 packets, 3650074 bytes [ReceivePcapFileThreadExitStats:source-pcap-file.c:387]
Suricata event types:
18 flow
7 fileinfo
4 http
2 dns
1 tls
1 stats
1 anomaly
rm logs/* -rf ; sudo suricata -S /dev/null -l logs/ -k none -r ce7ca983-9e4b-4251-a7c3-fefa3da02ebe.pcap --set "stream.reassembly.depth=2mb" ; echo "Suricata event types:" ; jq -r .event_type logs/eve.json | sort | uniq -c |sort -rn ; grep '"event_type":"alert"' logs/eve.json | jq .alert.signature | sort -rn | uniq -c | sort -rn ;
Notice: suricata: This is Suricata version 8.0.0-dev (13cc49388 2023-12-01) running in USER mode [LogVersion:suricata.c:1146]
Notice: threads: Threads created -> RX: 1 W: 16 FM: 1 FR: 1 Engine started. [TmThreadWaitOnThreadRunning:tm-threads.c:1891]
Notice: suricata: Signal Received. Stopping engine. [SuricataMainLoop:suricata.c:2805]
Notice: pcap: read 1 file, 5593 packets, 3650074 bytes [ReceivePcapFileThreadExitStats:source-pcap-file.c:387]
Suricata event types:
18 flow
8 fileinfo
4 http
2 dns
1 tls
1 stats
This is the anomaly event:
{
"timestamp": "2023-09-18T08:13:58.882971+0200",
"flow_id": 1676750216978051,
"pcap_cnt": 2878,
"event_type": "anomaly",
"src_ip": "192.168.100.237",
"src_port": 49175,
"dest_ip": "142.11.240.191",
"dest_port": 35361,
"proto": "TCP",
"pkt_src": "wire/pcap",
"tx_id": 3,
"anomaly": {
"app_proto": "http",
"type": "applayer",
"event": "UNABLE_TO_MATCH_RESPONSE_TO_REQUEST",
"layer": "proto_parser"
}
}
Updated by Philippe Antoine 2 months ago
- Status changed from New to Feedback
when i adjust the value to 2mb , I loose one anomaly event and gain one fileinfo event
That seems like an expected behavior, is it not ?