Project

General

Profile

Actions

Bug #5255

open
IK OD

output/alert: incorrect pcap_filename logged with --pcap-file-recursive

Bug #5255: output/alert: incorrect pcap_filename logged with --pcap-file-recursive

Added by Ivan Kachkivskyi almost 4 years ago. Updated 18 days ago.

Status:
In Review
Priority:
Normal
Assignee:
Target version:
Effort:
Difficulty:
Label:

Description

Bug has more discussion here: https://forum.suricata.io/t/pcap-filename-in-eve-json-is-not-accurate-when-using-pcap-file-continuous/558/

Reported pcap_filename in alerts are not correct.
pcap_filename is from a different event.
It is not possible to use suricata without --pcap-file-recursive because it takes too many time to scan each file separately. Suricata bootstrap goes too long (2 minutes per each file). There are hundreds of files each 10 minutes.
I have attached config file as you asked.
Please help me to resolve this issues.

Suricata version 7.0.0-dev (3a490fb16 2022-03-04)
Linux version 5.11.0-49-generic (buildd@lcy02-amd64-054) (gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0, GNU ld (GNU Binutils for Ubuntu) 2.36.1) #55-Ubuntu SMP Wed Jan 12 17:36:34 UTC 2022

/usr/bin/suricata -v -c /etc/suricata/suricata.yaml -r /opt/nids/test/ --pcap-file-recursive -l /var/log/suricata/ 2>/dev/null

{
timestamp: 2022-03-22T08:59:08.174415+0000,
flow_id: 1155331436357967,
pcap_cnt: 54854,
event_type: alert,
src_ip: 10.0.0.104,
src_port: 57621,
dest_ip: 10.0.0.255,
dest_port: 57621,
proto: UDP,
ether: {
src_mac: XX:XX:XX:4d:b2:4d,
dest_mac: ff:ff:ff:ff:ff:ff
},
community_id: 1:4nybWSscJYddpIgTcsmSA0dz7v4=,
alert: {
action: allowed,
gid: 1,
signature_id: 2027397,
rev: 1,
signature: ET POLICY Spotify P2P Client,
category: Not Suspicious Traffic,
severity: 3,
metadata: {
affected_product: [
Windows_Client_Apps
],
attack_target: [
Client_Endpoint
],
created_at: [
2019_05_30
],
deployment: [
Internal
],
performance_impact: [
Low
],
signature_severity: [
Minor
],
updated_at: [
2019_05_30
]
}
},
app_proto: failed,
flow: {
pkts_toserver: 1,
pkts_toclient: 0,
bytes_toserver: 86,
bytes_toclient: 0,
start: 2022-03-22T08:59:08.174415+0000
},
payload: hidden,
payload_printable: hidden,
stream: 0,
packet: hidden,
packet_info: {
linktype: 1
},
pcap_filename: /test//20220322090615350400-XX:XX:XX:XX:83:12-0e8816425288d9684f038bc55c3c03XX.pcap”
}
{
timestamp: 2022-03-22T09:04:08.198805+0000,
flow_id: 1155331436357967,
pcap_cnt: 118976,
event_type: alert,
src_ip: 10.0.0.104,
src_port: 57621,
dest_ip: 10.0.0.255,
dest_port: 57621,
proto: UDP,
ether: {
src_mac: XX:XX:XX:4d:b2:4d,
dest_mac: ff:ff:ff:ff:ff:ff
},
community_id: 1:4nybWSscJYddpIgTcsmSA0dz7v4=,
alert: {
action: allowed,
gid: 1,
signature_id: 2027397,
rev: 1,
signature: ET POLICY Spotify P2P Client,
category: Not Suspicious Traffic,
severity: 3,
metadata: {
affected_product: [
Windows_Client_Apps
],
attack_target: [
Client_Endpoint
],
created_at: [
2019_05_30
],
deployment: [
Internal
],
performance_impact: [
Low
],
signature_severity: [
Minor
],
updated_at: [
2019_05_30
]
}
},
app_proto: failed,
flow: {
pkts_toserver: 11,
pkts_toclient: 0,
bytes_toserver: 946,
bytes_toclient: 0,
start: 2022-03-22T08:59:08.174415+0000
},
payload: hidden,
payload_printable: hidden,
stream: 0,
packet: hidden,
packet_info: {
linktype: 1
},
pcap_filename: /test//20220322090914541400-XX:XX:XX:XX:2b:92-40b8c45e48e3fee6df38d2482ac16dXX.pcap”
}
{
timestamp: 2022-03-22T09:09:08.224668+0000,
flow_id: 1155331457946243,
pcap_cnt: 120400,
event_type: alert,
src_ip: 10.0.0.104,
src_port: 57621,
dest_ip: 10.0.0.255,
dest_port: 57621,
proto: UDP,
ether: {
src_mac: XX:XX:XX:4d:b2:4d,
dest_mac: ff:ff:ff:ff:ff:ff
},
community_id: 1:4nybWSscJYddpIgTcsmSA0dz7v4=,
alert: {
action: allowed,
gid: 1,
signature_id: 2027397,
rev: 1,
signature: ET POLICY Spotify P2P Client,
category: Not Suspicious Traffic,
severity: 3,
metadata: {
affected_product: [
Windows_Client_Apps
],
attack_target: [
Client_Endpoint
],
created_at: [
2019_05_30
],
deployment: [
Internal
],
performance_impact: [
Low
],
signature_severity: [
Minor
],
updated_at: [
2019_05_30
]
}
},
app_proto: failed,
flow: {
pkts_toserver: 10,
pkts_toclient: 0,
bytes_toserver: 860,
bytes_toclient: 0,
start: 2022-03-22T09:04:38.201347+0000
},
payload: hidden,
payload_printable: hidden,
stream: 0,
packet: hidden,
packet_info: {
linktype: 1
},
pcap_filename: /test//20220322090950822200-XX:XX:XX:XX:c1:c2-c550d07ceda9c9adfe0473975ab638XX.pcap”
}

Files

suricata.yaml (74.8 KB) suricata.yaml Suricata configuration file Ivan Kachkivskyi, 04/11/2022 07:34 AM

Subtasks 1 (1 open0 closed)

Bug #7780: output/alert: incorrect pcap_filename logged with --pcap-file-recursive (7.0.x backport)AssignedLukas SismisActions

IK Updated by Ivan Kachkivskyi almost 4 years ago Actions #1

  • Assignee deleted (OISF Dev)

VJ Updated by Victor Julien over 3 years ago Actions #2

  • Description updated (diff)

VJ Updated by Victor Julien over 3 years ago Actions #3

  • Priority changed from High to Normal

Whats the way to reproduce this?

IK Updated by Ivan Kachkivskyi over 3 years ago Actions #4

Please check Suricata forum links with configuration and files attached for steps to reproduce.
Link is on a top of message

IK Updated by Ivan Kachkivskyi over 3 years ago Actions #5

This is constantly reproducing. Try to take 10-20 pcap files from several computers and scan them several times with --pcap-file-recursive, you will see not correct behavior.

IK Updated by Ivan Kachkivskyi over 3 years ago Actions #6

Is there any update on this issue ?

PA Updated by Philippe Antoine almost 3 years ago Actions #7

  • Assignee set to OISF Dev

OD Updated by Ofer Dagan 10 months ago Actions #8

Hi,
I can confirm that this still happens with suricata 7.0.10. Are there any plans on fixing this?

LS Updated by Lukas Sismis 10 months ago Actions #9

  • Assignee changed from OISF Dev to Lukas Sismis

LS Updated by Lukas Sismis 10 months ago Actions #10

  • Target version changed from TBD to 8.0.0
  • Label Needs backport to 7.0 added

LS Updated by Lukas Sismis 10 months ago Actions #11

Suspected race condition when the PCAP file name is stored in a global variable while some workers might still process the previous PCAP

OT Updated by OISF Ticketbot 10 months ago Actions #12

  • Subtask #7780 added

OT Updated by OISF Ticketbot 10 months ago Actions #13

  • Label deleted (Needs backport to 7.0)

LS Updated by Lukas Sismis 9 months ago Actions #14

  • Status changed from New to Assigned

VJ Updated by Victor Julien 9 months ago Actions #15

  • Target version changed from 8.0.0 to 9.0.0-beta1

LS Updated by Lukas Sismis 8 months ago Actions #16

  • Status changed from Assigned to In Progress

LS Updated by Lukas Sismis 8 months ago Actions #17

  • Status changed from In Progress to In Review

OD Updated by Ofer Dagan 6 months ago Actions #18

Hi there,
Is there an update regarding this ticket?

LS Updated by Lukas Sismis 6 months ago Actions #19

SB Updated by Shivani Bhardwaj 5 months ago Actions #20

  • Subject changed from Reported pcap_filename in alerts are not correct to output/alert: incorrect pcap_filename logged with --pcap-file-recursive

VJ Updated by Victor Julien about 1 month ago Actions #21

I think the work in #7786 can serve as the basis for this. There is now a ref counted pointer to the current file in the packet, I think. Would you agree @Ofer Dagan ?

OD Updated by Ofer Dagan about 1 month ago Actions #22

Yes I think it will be a very clean solution. It will be really simple to get to it in `OutputJsonBuilderBuffer` - `p->pcap_v->pfv->filename`. Thanks to the ref counters we added, this struct will stay alive.
Note that the path for the stats event will stay with the old function but I don't think it's really a problem there.
This should be a really simple fix. Let me know if you want me to open the PR myself @Victor Julien @Lukas Sismis

VJ Updated by Victor Julien about 1 month ago Actions #23

  • Status changed from In Review to Assigned
  • Assignee changed from Lukas Sismis to Ofer Dagan

@Ofer Dagan that would be great, thanks!

OD Updated by Ofer Dagan about 1 month ago Actions #24

@Victor Julien Note this found an error in the previous PR! I'm thinking of adding the fix as a separate commit in the PR I'm opening for this one. It's not something we can detect on a suricata-verify test (only UTs) and it's blocking this solution so I believe it's fine to be on the same PR. Do I need to also open a ticket for that or because it hadn't released yet it's fine?
Let me know how you prefer to proceed here.

PA Updated by Philippe Antoine 18 days ago Actions #26

  • Status changed from Assigned to In Review
Actions

Also available in: PDF Atom