Bug #5255
openoutput/alert: incorrect pcap_filename logged with --pcap-file-recursive
Description
Bug has more discussion here: https://forum.suricata.io/t/pcap-filename-in-eve-json-is-not-accurate-when-using-pcap-file-continuous/558/
Reported pcap_filename in alerts are not correct.
pcap_filename is from a different event.
It is not possible to use suricata without --pcap-file-recursive because it takes too many time to scan each file separately. Suricata bootstrap goes too long (2 minutes per each file). There are hundreds of files each 10 minutes.
I have attached config file as you asked.
Please help me to resolve this issues.
Suricata version 7.0.0-dev (3a490fb16 2022-03-04) Linux version 5.11.0-49-generic (buildd@lcy02-amd64-054) (gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0, GNU ld (GNU Binutils for Ubuntu) 2.36.1) #55-Ubuntu SMP Wed Jan 12 17:36:34 UTC 2022 /usr/bin/suricata -v -c /etc/suricata/suricata.yaml -r /opt/nids/test/ --pcap-file-recursive -l /var/log/suricata/ 2>/dev/null
{
“timestamp”: “2022-03-22T08:59:08.174415+0000”,
“flow_id”: 1155331436357967,
“pcap_cnt”: 54854,
“event_type”: “alert”,
“src_ip”: “10.0.0.104”,
“src_port”: 57621,
“dest_ip”: “10.0.0.255”,
“dest_port”: 57621,
“proto”: “UDP”,
“ether”: {
“src_mac”: “XX:XX:XX:4d:b2:4d”,
“dest_mac”: “ff:ff:ff:ff:ff:ff”
},
“community_id”: “1:4nybWSscJYddpIgTcsmSA0dz7v4=”,
“alert”: {
“action”: “allowed”,
“gid”: 1,
“signature_id”: 2027397,
“rev”: 1,
“signature”: “ET POLICY Spotify P2P Client”,
“category”: “Not Suspicious Traffic”,
“severity”: 3,
“metadata”: {
“affected_product”: [
“Windows_Client_Apps”
],
“attack_target”: [
“Client_Endpoint”
],
“created_at”: [
“2019_05_30”
],
“deployment”: [
“Internal”
],
“performance_impact”: [
“Low”
],
“signature_severity”: [
“Minor”
],
“updated_at”: [
“2019_05_30”
]
}
},
“app_proto”: “failed”,
“flow”: {
“pkts_toserver”: 1,
“pkts_toclient”: 0,
“bytes_toserver”: 86,
“bytes_toclient”: 0,
“start”: “2022-03-22T08:59:08.174415+0000”
},
“payload”: “hidden”,
“payload_printable”: “hidden”,
“stream”: 0,
“packet”: “hidden”,
“packet_info”: {
“linktype”: 1
},
“pcap_filename”: “/test//20220322090615350400-XX:XX:XX:XX:83:12-0e8816425288d9684f038bc55c3c03XX.pcap”
}
{
“timestamp”: “2022-03-22T09:04:08.198805+0000”,
“flow_id”: 1155331436357967,
“pcap_cnt”: 118976,
“event_type”: “alert”,
“src_ip”: “10.0.0.104”,
“src_port”: 57621,
“dest_ip”: “10.0.0.255”,
“dest_port”: 57621,
“proto”: “UDP”,
“ether”: {
“src_mac”: “XX:XX:XX:4d:b2:4d”,
“dest_mac”: “ff:ff:ff:ff:ff:ff”
},
“community_id”: “1:4nybWSscJYddpIgTcsmSA0dz7v4=”,
“alert”: {
“action”: “allowed”,
“gid”: 1,
“signature_id”: 2027397,
“rev”: 1,
“signature”: “ET POLICY Spotify P2P Client”,
“category”: “Not Suspicious Traffic”,
“severity”: 3,
“metadata”: {
“affected_product”: [
“Windows_Client_Apps”
],
“attack_target”: [
“Client_Endpoint”
],
“created_at”: [
“2019_05_30”
],
“deployment”: [
“Internal”
],
“performance_impact”: [
“Low”
],
“signature_severity”: [
“Minor”
],
“updated_at”: [
“2019_05_30”
]
}
},
“app_proto”: “failed”,
“flow”: {
“pkts_toserver”: 11,
“pkts_toclient”: 0,
“bytes_toserver”: 946,
“bytes_toclient”: 0,
“start”: “2022-03-22T08:59:08.174415+0000”
},
“payload”: “hidden”,
“payload_printable”: “hidden”,
“stream”: 0,
“packet”: “hidden”,
“packet_info”: {
“linktype”: 1
},
“pcap_filename”: “/test//20220322090914541400-XX:XX:XX:XX:2b:92-40b8c45e48e3fee6df38d2482ac16dXX.pcap”
}
{
“timestamp”: “2022-03-22T09:09:08.224668+0000”,
“flow_id”: 1155331457946243,
“pcap_cnt”: 120400,
“event_type”: “alert”,
“src_ip”: “10.0.0.104”,
“src_port”: 57621,
“dest_ip”: “10.0.0.255”,
“dest_port”: 57621,
“proto”: “UDP”,
“ether”: {
“src_mac”: “XX:XX:XX:4d:b2:4d”,
“dest_mac”: “ff:ff:ff:ff:ff:ff”
},
“community_id”: “1:4nybWSscJYddpIgTcsmSA0dz7v4=”,
“alert”: {
“action”: “allowed”,
“gid”: 1,
“signature_id”: 2027397,
“rev”: 1,
“signature”: “ET POLICY Spotify P2P Client”,
“category”: “Not Suspicious Traffic”,
“severity”: 3,
“metadata”: {
“affected_product”: [
“Windows_Client_Apps”
],
“attack_target”: [
“Client_Endpoint”
],
“created_at”: [
“2019_05_30”
],
“deployment”: [
“Internal”
],
“performance_impact”: [
“Low”
],
“signature_severity”: [
“Minor”
],
“updated_at”: [
“2019_05_30”
]
}
},
“app_proto”: “failed”,
“flow”: {
“pkts_toserver”: 10,
“pkts_toclient”: 0,
“bytes_toserver”: 860,
“bytes_toclient”: 0,
“start”: “2022-03-22T09:04:38.201347+0000”
},
“payload”: “hidden”,
“payload_printable”: “hidden”,
“stream”: 0,
“packet”: “hidden”,
“packet_info”: {
“linktype”: 1
},
“pcap_filename”: “/test//20220322090950822200-XX:XX:XX:XX:c1:c2-c550d07ceda9c9adfe0473975ab638XX.pcap”
}
Files
IK Updated by Ivan Kachkivskyi over 4 years ago
- Assignee deleted (
OISF Dev)
VJ Updated by Victor Julien almost 4 years ago
- Description updated (diff)
VJ Updated by Victor Julien almost 4 years ago
- Priority changed from High to Normal
Whats the way to reproduce this?
IK Updated by Ivan Kachkivskyi almost 4 years ago
Please check Suricata forum links with configuration and files attached for steps to reproduce.
Link is on a top of message
IK Updated by Ivan Kachkivskyi almost 4 years ago
This is constantly reproducing. Try to take 10-20 pcap files from several computers and scan them several times with --pcap-file-recursive, you will see not correct behavior.
IK Updated by Ivan Kachkivskyi almost 4 years ago
Is there any update on this issue ?
PA Updated by Philippe Antoine about 3 years ago
- Assignee set to OISF Dev
OD Updated by Ofer Dagan about 1 year ago
Hi,
I can confirm that this still happens with suricata 7.0.10. Are there any plans on fixing this?
LS Updated by Lukas Sismis about 1 year ago
- Assignee changed from OISF Dev to Lukas Sismis
LS Updated by Lukas Sismis about 1 year ago
- Target version changed from TBD to 8.0.0
- Label Needs backport to 7.0 added
LS Updated by Lukas Sismis about 1 year ago
Suspected race condition when the PCAP file name is stored in a global variable while some workers might still process the previous PCAP
OT Updated by OISF Ticketbot about 1 year ago
- Subtask #7780 added
OT Updated by OISF Ticketbot about 1 year ago
- Label deleted (
Needs backport to 7.0)
LS Updated by Lukas Sismis about 1 year ago
- Status changed from New to Assigned
VJ Updated by Victor Julien about 1 year ago
- Target version changed from 8.0.0 to 9.0.0-beta1
LS Updated by Lukas Sismis about 1 year ago
- Status changed from Assigned to In Progress
LS Updated by Lukas Sismis about 1 year ago
- Status changed from In Progress to In Review
OD Updated by Ofer Dagan 10 months ago
Hi there,
Is there an update regarding this ticket?
LS Updated by Lukas Sismis 10 months ago
On-hold to resume later
SB Updated by Shivani Bhardwaj 9 months ago
- Subject changed from Reported pcap_filename in alerts are not correct to output/alert: incorrect pcap_filename logged with --pcap-file-recursive
VJ Updated by Victor Julien 5 months ago
I think the work in #7786 can serve as the basis for this. There is now a ref counted pointer to the current file in the packet, I think. Would you agree @Ofer Dagan ?
OD Updated by Ofer Dagan 5 months ago
Yes I think it will be a very clean solution. It will be really simple to get to it in `OutputJsonBuilderBuffer` - `p->pcap_v->pfv->filename`. Thanks to the ref counters we added, this struct will stay alive.
Note that the path for the stats event will stay with the old function but I don't think it's really a problem there.
This should be a really simple fix. Let me know if you want me to open the PR myself @Victor Julien @Lukas Sismis
VJ Updated by Victor Julien 5 months ago
- Status changed from In Review to Assigned
- Assignee changed from Lukas Sismis to Ofer Dagan
@Ofer Dagan that would be great, thanks!
OD Updated by Ofer Dagan 5 months ago
@Victor Julien Note this found an error in the previous PR! I'm thinking of adding the fix as a separate commit in the PR I'm opening for this one. It's not something we can detect on a suricata-verify test (only UTs) and it's blocking this solution so I believe it's fine to be on the same PR. Do I need to also open a ticket for that or because it hadn't released yet it's fine?
Let me know how you prefer to proceed here.
OD Updated by Ofer Dagan 5 months ago
Opened a PR - https://github.com/OISF/suricata/pull/14857
PA Updated by Philippe Antoine 4 months ago
- Status changed from Assigned to In Review
PA Updated by Philippe Antoine about 1 month ago
Current PR https://github.com/OISF/suricata/pull/15176