Project

General

Profile

Actions

Bug #6347

closed

log-pcap: crash with suricata.yaml setting max-file to 1

Added by kun xu 8 months ago. Updated 3 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
low
Label:

Description

Server status:
  • I use logrotate to split the 'eve.json' file at 23:59 every day, and at the same time, I use 'sharedscripts' to gracefully restart the Suricata service.
    [root@Network-Security-Event-Validation-System ~]# cat /home/Suricata/log/suricata/suricata-logrotate 
    /home/Suricata/log/suricata/eve.json {
    missingok
        daily
        create 0666 root root
        dateext
        rotate 999999
        sharedscripts
        postrotate
            service suricata restart
        endscript
    }
    
    [root@Network-Security-Event-Validation-System ~]# cat /etc/crontab 
    SHELL=/bin/bash
    PATH=/sbin:/bin:/usr/sbin:/usr/bin
    MAILTO=root
    
    # For details see man 4 crontabs
    
    # Example of job definition:
    # .---------------- minute (0 - 59)
    # |  .------------- hour (0 - 23)
    # |  |  .---------- day of month (1 - 31)
    # |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
    # |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
    # |  |  |  |  |
    # *  *  *  *  * user-name  command to be executed
    
      59 23  *  *  * root logrotate -vf /home/Suricata/log/suricata/suricata-logrotate
    
    [root@Network-Security-Event-Validation-System ~]# cat /usr/lib/systemd/system/suricata.service
    [Unit]
    Description=Suricata
    
    [Service]
    ExecStart=/home/Suricata/bin/suricata --af-packet -vv
    
    [Install]
    WantedBy=multi-user.target
    
    [root@Network-Security-Event-Validation-System ~]# /home/Suricata/bin/suricata -V
    This is Suricata version 7.0.0 RELEASE
    
  • In order to facilitate packet filtering, I hope that suricata will only save one pcap file every day, so set the value of max-files in suricata.yaml to 1.
    - pcap-log:
          enabled: yes
          filename: alert.pcap
    
          # File size limit.  Can be specified in kb, mb, gb.  Just a number
          # is parsed as bytes.
          limit: 100gb
    
          # If set to a value, ring buffer mode is enabled. Will keep maximum of
          # "max-files" of size "limit" 
          max-files: 1
    
          # Compression algorithm for pcap files. Possible values: none, lz4.
          # Enabling compression is incompatible with the sguil mode. Note also
          # that on Windows, enabling compression will *increase* disk I/O.
          compression: none
    
          # Further options for lz4 compression. The compression level can be set
          # to a value between 0 and 16, where higher values result in higher
          # compression.
          #lz4-checksum: no
          #lz4-level: 0
    
          mode: sguil # normal, multi or sguil.
    
          # Directory to place pcap files. If not provided the default log
          # directory will be used. Required for "sguil" mode.
          dir: /home/pcap-log
    
Fault description:
  • After complete log cutting and restart of the suricata service at 23:59 every day, the main process of suricata will crash abnormally.
    9月 10 23:59:17 Network-Security-Event-Validation-System suricata[1545]: [1561] Perf: af-packet: em2: setting socket buffer to 2147483647
    9月 10 23:59:17 Network-Security-Event-Validation-System suricata[1545]: [1561] Perf: af-packet: em2: rx ring params: block_size=1048576 block_nr=
    9月 10 23:59:17 Network-Security-Event-Validation-System suricata[1545]: [1564] Perf: af-packet: em2: setting socket buffer to 2147483647
    9月 10 23:59:17 Network-Security-Event-Validation-System suricata[1545]: [1564] Perf: af-packet: em2: rx ring params: block_size=1048576 block_nr=
    9月 10 23:59:17 Network-Security-Event-Validation-System suricata[1545]: [1566] Perf: af-packet: em2: setting socket buffer to 2147483647
    9月 10 23:59:17 Network-Security-Event-Validation-System suricata[1545]: [1566] Perf: af-packet: em2: rx ring params: block_size=1048576 block_nr=
    9月 10 23:59:18 Network-Security-Event-Validation-System suricata[1545]: [1567] Perf: af-packet: em2: setting socket buffer to 2147483647
    9月 10 23:59:18 Network-Security-Event-Validation-System suricata[1545]: [1567] Perf: af-packet: em2: rx ring params: block_size=1048576 block_nr=
    9月 10 23:59:18 Network-Security-Event-Validation-System suricata[1545]: [1568] Perf: af-packet: em4: setting socket buffer to 2147483647
    9月 10 23:59:18 Network-Security-Event-Validation-System suricata[1545]: [1568] Perf: af-packet: em4: rx ring params: block_size=1048576 block_nr=
    9月 10 23:59:18 Network-Security-Event-Validation-System suricata[1545]: [1569] Perf: af-packet: em4: setting socket buffer to 2147483647
    9月 10 23:59:18 Network-Security-Event-Validation-System suricata[1545]: [1569] Perf: af-packet: em4: rx ring params: block_size=1048576 block_nr=
    9月 10 23:59:18 Network-Security-Event-Validation-System suricata[1545]: [1570] Perf: af-packet: em4: setting socket buffer to 2147483647
    9月 10 23:59:18 Network-Security-Event-Validation-System suricata[1545]: [1570] Perf: af-packet: em4: rx ring params: block_size=1048576 block_nr=
    9月 10 23:59:18 Network-Security-Event-Validation-System suricata[1545]: [1571] Perf: af-packet: em4: setting socket buffer to 2147483647
    9月 10 23:59:18 Network-Security-Event-Validation-System suricata[1545]: [1571] Perf: af-packet: em4: rx ring params: block_size=1048576 block_nr=
    9月 10 23:59:18 Network-Security-Event-Validation-System suricata[1545]: [1572] Perf: af-packet: em4: setting socket buffer to 2147483647
    9月 10 23:59:18 Network-Security-Event-Validation-System suricata[1545]: [1572] Perf: af-packet: em4: rx ring params: block_size=1048576 block_nr=
    9月 10 23:59:18 Network-Security-Event-Validation-System suricata[1545]: [1574] Perf: af-packet: em4: setting socket buffer to 2147483647
    9月 10 23:59:18 Network-Security-Event-Validation-System suricata[1545]: [1574] Perf: af-packet: em4: rx ring params: block_size=1048576 block_nr=
    9月 10 23:59:18 Network-Security-Event-Validation-System suricata[1545]: [1545] Notice: threads: Threads created -> W: 12 FM: 1 FR: 1   Engine sta
    9月 11 00:00:33 Network-Security-Event-Validation-System systemd[1]: suricata.service: main process exited, code=killed, status=11/SEGV
    9月 11 00:00:33 Network-Security-Event-Validation-System systemd[1]: Unit suricata.service entered failed state.
    9月 11 00:00:33 Network-Security-Event-Validation-System systemd[1]: suricata.service failed.
    
  • Check the messages log information and find the following error segfault at 8 ip 0000000000682e3a sp 00007f1cbb8390e0 error 4 in suricata[400000+995000]
    Sep 10 23:59:18 Network-Security-Event-Validation-System suricata: [1567] Perf: af-packet: em2: rx ring params: block_size=1048576 block_nr=155 fr
    ame_size=1616 frame_nr=100440 (mem: 162529280)
    Sep 10 23:59:18 Network-Security-Event-Validation-System suricata: [1568] Perf: af-packet: em4: setting socket buffer to 2147483647
    Sep 10 23:59:18 Network-Security-Event-Validation-System kernel: device em4 entered promiscuous mode
    Sep 10 23:59:18 Network-Security-Event-Validation-System suricata: [1568] Perf: af-packet: em4: rx ring params: block_size=1048576 block_nr=155 fr
    ame_size=1616 frame_nr=100440 (mem: 162529280)
    Sep 10 23:59:18 Network-Security-Event-Validation-System suricata: [1569] Perf: af-packet: em4: setting socket buffer to 2147483647
    Sep 10 23:59:18 Network-Security-Event-Validation-System suricata: [1569] Perf: af-packet: em4: rx ring params: block_size=1048576 block_nr=155 fr
    ame_size=1616 frame_nr=100440 (mem: 162529280)
    Sep 10 23:59:18 Network-Security-Event-Validation-System suricata: [1570] Perf: af-packet: em4: setting socket buffer to 2147483647
    Sep 10 23:59:18 Network-Security-Event-Validation-System suricata: [1570] Perf: af-packet: em4: rx ring params: block_size=1048576 block_nr=155 fr
    ame_size=1616 frame_nr=100440 (mem: 162529280)
    Sep 10 23:59:18 Network-Security-Event-Validation-System suricata: [1571] Perf: af-packet: em4: setting socket buffer to 2147483647
    Sep 10 23:59:18 Network-Security-Event-Validation-System suricata: [1571] Perf: af-packet: em4: rx ring params: block_size=1048576 block_nr=155 fr
    ame_size=1616 frame_nr=100440 (mem: 162529280)
    Sep 10 23:59:18 Network-Security-Event-Validation-System suricata: [1572] Perf: af-packet: em4: setting socket buffer to 2147483647
    Sep 10 23:59:18 Network-Security-Event-Validation-System suricata: [1572] Perf: af-packet: em4: rx ring params: block_size=1048576 block_nr=155 fr
    ame_size=1616 frame_nr=100440 (mem: 162529280)
    Sep 10 23:59:18 Network-Security-Event-Validation-System suricata: [1574] Perf: af-packet: em4: setting socket buffer to 2147483647
    Sep 10 23:59:18 Network-Security-Event-Validation-System suricata: [1574] Perf: af-packet: em4: rx ring params: block_size=1048576 block_nr=155 fr
    ame_size=1616 frame_nr=100440 (mem: 162529280)
    Sep 10 23:59:18 Network-Security-Event-Validation-System suricata: [1545] Notice: threads: Threads created -> W: 12 FM: 1 FR: 1   Engine started.
    Sep 11 00:00:09 Network-Security-Event-Validation-System kernel: W#03-em4[1570]: segfault at 8 ip 0000000000682e3a sp 00007f1cbb8390e0 error 4 in 
    suricata[400000+995000]
    Sep 11 00:00:33 Network-Security-Event-Validation-System kernel: device em2 left promiscuous mode
    Sep 11 00:00:33 Network-Security-Event-Validation-System kernel: device em4 left promiscuous mode
    Sep 11 00:00:33 Network-Security-Event-Validation-System systemd: suricata.service: main process exited, code=killed, status=11/SEGV
    Sep 11 00:00:33 Network-Security-Event-Validation-System systemd: Unit suricata.service entered failed state.
    Sep 11 00:00:33 Network-Security-Event-Validation-System systemd: suricata.service failed.
    
  • Use the dump file to view the error information as follows:
    [root@Network-Security-Event-Validation-System ~]# gdb /home/Suricata/bin/suricata /core.1545   
    GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7
    Copyright (C) 2013 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.  Type "show copying" 
    and "show warranty" for details.
    This GDB was configured as "x86_64-redhat-linux-gnu".
    For bug reporting instructions, please see:
    <http://www.gnu.org/software/gdb/bugs/>...
    Reading symbols from /home/Suricata/bin/suricata...done.
    
    warning: core file may not match specified executable file.
    [New LWP 1570]
    [New LWP 1575]
    [New LWP 1564]
    [New LWP 1577]
    [New LWP 1545]
    [New LWP 1576]
    [New LWP 1578]
    [New LWP 1567]
    [New LWP 1579]
    [New LWP 1572]
    [New LWP 1559]
    [New LWP 1566]
    [New LWP 1569]
    [New LWP 1561]
    [New LWP 1560]
    [New LWP 1571]
    [New LWP 1574]
    [New LWP 1568]
    [Thread debugging using libthread_db enabled]
    Using host libthread_db library "/lib64/libthread_db.so.1".
    Core was generated by `/home/Suricata/bin/suricata --af-packet -vv'.
    Program terminated with signal 11, Segmentation fault.
    #0  0x0000000000682e3a in PcapLogRotateFile (pl=pl@entry=0x27492160, t=<optimized out>) at log-pcap.c:375
    375     log-pcap.c: 没有那个文件或目录.
    Missing separate debuginfos, use: debuginfo-install elfutils-libelf-0.176-5.el7.x86_64 file-libs-5.11-37.el7.x86_64 glibc-2.17-326.el7_9.x86_64 jansson-2.10-1.el7.x86_64 libcap-ng-0.7.5-4.el7.x86_64 libgcc-4.8.5-44.el7.x86_64 libnet-1.1.6-7.el7.x86_64 libpcap-1.5.3-12.el7.x86_64 libyaml-0.1.4-11.el7_0.x86_64 lz4-1.8.3-1.el7.x86_64 pcre2-10.23-2.el7.x86_64 zlib-1.2.7-21.el7_9.x86_64
    
  • Change the value of max-files in suricata.yaml to 100 and the fault will no longer occur.
Help:
  • If I only want to save one pcap file every day, what should be the correct configuration?

Files

suricata.yaml (68 KB) suricata.yaml kun xu, 09/13/2023 02:53 AM
messages (591 KB) messages kun xu, 09/13/2023 02:53 AM

Subtasks 1 (0 open1 closed)

Bug #6688: log-pcap: crash with suricata.yaml setting max-file to 1 (7.0.x backport)ClosedJeff LucovskyActions
Actions #1

Updated by Victor Julien 8 months ago

We should probably remove the "sguil" mode.

Actions #2

Updated by Victor Julien 8 months ago

  • Subject changed from Suricata crashes when suricata.yaml setting max-file to 1 in pcap-log config to log-pcap: crash with suricata.yaml setting max-file to 1
Actions #3

Updated by Jeff Lucovsky 8 months ago

  • Assignee changed from Shivani Bhardwaj to Jeff Lucovsky
Actions #4

Updated by Victor Julien 4 months ago

  • Status changed from New to Assigned
  • Target version changed from TBD to 7.0.4
Actions #5

Updated by Jeff Lucovsky 4 months ago

  • Status changed from Assigned to In Review
Per @Victor Julien:
  • sguil mode will be removed from 8.0.x
  • squil mode usage will force deprecation message in 7.0.x
Actions #7

Updated by Jeff Lucovsky 4 months ago

  • Label Needs backport to 7.0 added
Actions #8

Updated by Jeff Lucovsky 4 months ago

  • Target version changed from 7.0.4 to 8.0.0-beta1
Actions #9

Updated by OISF Ticketbot 4 months ago

  • Subtask #6688 added
Actions #10

Updated by OISF Ticketbot 4 months ago

  • Label deleted (Needs backport to 7.0)
Actions #11

Updated by Jeff Lucovsky 4 months ago

  • Label Needs backport to 7.0 added
Actions #12

Updated by OISF Ticketbot 4 months ago

  • Label deleted (Needs backport to 7.0)
Actions #13

Updated by Jeff Lucovsky 3 months ago

  • Status changed from In Review to Closed
Actions

Also available in: Atom PDF