Bug #1174
Updated by Victor Julien about 10 years ago
I'm having a segfault occur about once a week with suricata 2.0 . I think the issue is may not be specific to just 2.0, we ran 1.4.7 for a little while and it segfaulted once or twice too. All the core dumps I've captured seem to point at a buffer overflow in the memcpy function called at stream-tcp-reassemble.c line 3139. <pre> Stack trace: (gdb) bt #0 0x0000003968432925 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #1 0x0000003968434105 in abort () at abort.c:92 #2 0x0000003968470837 in __libc_message (do_abort=2, fmt=0x3968557930 "*** %s ***: %s terminated\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:198 #3 0x0000003968502827 in __fortify_fail ( msg=0x39685578d6 "buffer overflow detected") at fortify_fail.c:32 #4 0x0000003968500710 in __chk_fail () at chk_fail.c:29 #5 0x0000000000511230 in memcpy (tv=0xad3dd80, ra_ctx=0x7f75c0000fb0, ssn=0x7f75c3ae0050, stream=0x7f75c3ae0058, p=0x33e4230) at /usr/include/bits/string3.h:52 #6 StreamTcpReassembleAppLayer (tv=0xad3dd80, ra_ctx=0x7f75c0000fb0, ssn=0x7f75c3ae0050, stream=0x7f75c3ae0058, p=0x33e4230) at stream-tcp-reassemble.c:3139 #7 0x00000000005115c0 in StreamTcpReassembleHandleSegmentUpdateACK ( tv=0xad3dd80, ra_ctx=0x7f75c0000fb0, ssn=0x7f75c3ae0050, stream=0x7f75c3ae0058, p=0x33e4230) at stream-tcp-reassemble.c:3545 #8 0x0000000000513773 in StreamTcpReassembleHandleSegment (tv=0xad3dd80, ra_ctx=0x7f75c0000fb0, ssn=0x7f75c3ae0050, stream=0x7f75c3ae00a0, p=0x33e4230, pq=<value optimized out>) at stream-tcp-reassemble.c:3573 #9 0x000000000050b09b in HandleEstablishedPacketToClient (tv=0xad3dd80, p=0x33e4230, stt=0x7f75c00008c0, ssn=0x7f75c3ae0050, pq=<value optimized out>) at stream-tcp.c:2091 #10 StreamTcpPacketStateEstablished (tv=0xad3dd80, p=0x33e4230, stt=0x7f75c00008c0, ssn=0x7f75c3ae0050, pq=<value optimized out>) at stream-tcp.c:2337 #11 0x000000000050e670 in StreamTcpPacket (tv=0xad3dd80, p=0x33e4230, stt=0x7f75c00008c0, pq=0xad3deb0) at stream-tcp.c:4243 #12 0x000000000050f4d3 in StreamTcp (tv=0xad3dd80, p=0x33e4230, data=0x7f75c00008c0, pq=<value optimized out>, postpq=<value optimized out>) at stream-tcp.c:4485 #13 0x0000000000524109 in TmThreadsSlotVarRun (tv=0xad3dd80, p=0x33e4230, slot=<value optimized out>) at tm-threads.c:557 #14 0x00000000005242e9 in TmThreadsSlotVar (td=0xad3dd80) at tm-threads.c:814 #15 0x0000003aede079d1 in start_thread (arg=0x7f75cbfff700) at pthread_create.c:301 #16 0x00000039684e8b6d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 </pre> compiled with command: <pre> CFLAGS="-O2 -g" CCFLAGS="-O2 -g" ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --libdir=/usr/lib64 --enable-gccprotect --with-nss-includes=/usr/include/nss3 --with-libnspr-includes=/usr/include/nspr </pre> <pre> Suricata Configuration: AF_PACKET support: yes PF_RING support: no NFQueue support: no IPFW support: no DAG enabled: no Napatech enabled: no Unix socket enabled: yes Detection enabled: yes libnss support: yes libnspr support: yes libjansson support: yes Prelude support: no PCRE jit: no libluajit: no libgeoip: no Non-bundled htp: no Old barnyard2 support: no CUDA enabled: no Suricatasc install: yes Unit tests enabled: no Debug output enabled: no Debug validation enabled: no Profiling enabled: no Profiling locks enabled: no Coccinelle / spatch: no Generic build parameters: Installation prefix (--prefix): /usr Configuration directory (--sysconfdir): /etc/suricata/ Log directory (--localstatedir) : /var/log/suricata/ Host: x86_64-unknown-linux-gnu GCC binary: gcc GCC Protect enabled: yes GCC march native enabled: yes GCC Profile enabled: no </pre> Suricata run with command: suricata -c /etc/suricata/suricata.yaml --af-packet=eth2 -D suricata.yaml minified: <pre> %YAML 1.1 --- host-mode: sniffer-only default-log-dir: /var/log/suricata/ unix-command: enabled: no outputs: - fast: enabled: no filename: fast.log append: yes - eve-log: enabled: no type: file #file|syslog|unix_dgram|unix_stream filename: eve.json types: - alert - http: extended: yes # enable this for extended logging information - dns - tls: extended: yes # enable this for extended logging information - files: force-magic: no # force logging magic on all logged files force-md5: no # force logging of md5 checksums - ssh - unified2-alert: enabled: yes filename: unified2.alert limit: 32mb sensor-id: 0 xff: enabled: yes mode: extra-data header: X-Forwarded-For - http-log: enabled: no filename: http.log append: yes - tls-log: enabled: no # Log TLS connections. filename: tls.log # File to store TLS logs. append: yes certs-log-dir: certs # directory to store the certificates files - dns-log: enabled: no filename: dns.log append: yes - pcap-info: enabled: no - pcap-log: enabled: no filename: log.pcap limit: 1000mb max-files: 2000 mode: normal # normal or sguil. use-stream-depth: no #If set to "yes" packets seen after reaching stream inspection depth are ignored. "no" logs all packets - alert-debug: enabled: no filename: alert-debug.log append: yes - alert-prelude: enabled: no profile: suricata log-packet-content: no log-packet-header: yes - stats: enabled: no filename: stats.log interval: 8 - syslog: enabled: no facility: local5 - drop: enabled: no filename: drop.log append: yes - file-store: enabled: no # set to yes to enable log-dir: files # directory to store the files force-magic: no # force logging magic on all stored files force-md5: no # force logging of md5 checksums - file-log: enabled: no filename: files-json.log append: yes force-magic: no # force logging magic on all logged files force-md5: no # force logging of md5 checksums magic-file: /usr/share/file/magic nfq: af-packet: - interface: eth2 threads: 8 cluster-id: 99 cluster-type: cluster_flow defrag: yes use-mmap: no checksum-checks: no - interface: eth1 threads: 1 cluster-id: 98 cluster-type: cluster_flow defrag: yes - interface: default legacy: uricontent: enabled detect-engine: - profile: high - custom-values: toclient-src-groups: 15 toclient-dst-groups: 15 toclient-sp-groups: 15 toclient-dp-groups: 20 toserver-src-groups: 15 toserver-dst-groups: 15 toserver-sp-groups: 15 toserver-dp-groups: 40 - sgh-mpm-context: auto - inspection-recursion-limit: 3000 threading: set-cpu-affinity: no cpu-affinity: - management-cpu-set: cpu: [ 0 ] # include only these cpus in affinity settings - receive-cpu-set: cpu: [ 0 ] # include only these cpus in affinity settings - decode-cpu-set: cpu: [ 0, 1 ] mode: "balanced" - stream-cpu-set: cpu: [ "0-1" ] - detect-cpu-set: cpu: [ "all" ] mode: "exclusive" # run detect threads in these cpus prio: low: [ 0 ] medium: [ "1-2" ] high: [ 3 ] default: "medium" - verdict-cpu-set: cpu: [ 0 ] prio: default: "high" - reject-cpu-set: cpu: [ 0 ] prio: default: "low" - output-cpu-set: cpu: [ "all" ] prio: default: "medium" detect-thread-ratio: 1.5 cuda: mpm: data-buffer-size-min-limit: 0 data-buffer-size-max-limit: 1500 cudabuffer-buffer-size: 500mb gpu-transfer-size: 50mb batching-timeout: 2000 device-id: 0 cuda-streams: 2 mpm-algo: ac pattern-matcher: - b2gc: search-algo: B2gSearchBNDMq hash-size: low bf-size: medium - b2gm: search-algo: B2gSearchBNDMq hash-size: low bf-size: medium - b2g: search-algo: B2gSearchBNDMq hash-size: low bf-size: medium - b3g: search-algo: B3gSearchBNDMq hash-size: low bf-size: medium - wumanber: hash-size: low bf-size: medium defrag: memcap: 32mb hash-size: 65536 trackers: 65535 # number of defragmented flows to follow max-frags: 65535 # number of fragments to keep (higher than trackers) prealloc: yes timeout: 60 flow: memcap: 64mb hash-size: 65536 prealloc: 10000 emergency-recovery: 30 vlan: use-for-tracking: true flow-timeouts: default: new: 30 established: 300 closed: 0 emergency-new: 10 emergency-established: 100 emergency-closed: 0 tcp: new: 60 established: 3600 closed: 120 emergency-new: 10 emergency-established: 300 emergency-closed: 20 udp: new: 30 established: 300 emergency-new: 10 emergency-established: 100 icmp: new: 30 established: 300 emergency-new: 10 emergency-established: 100 stream: memcap: 32mb checksum-validation: no # reject wrong csums inline: auto # auto will use inline mode in IPS mode, yes or no set it statically reassembly: memcap: 128mb depth: 1mb # reassemble 1mb into a stream toserver-chunk-size: 2560 toclient-chunk-size: 2560 randomize-chunk-size: yes host: hash-size: 4096 prealloc: 1000 memcap: 16777216 logging: default-log-level: notice default-output-filter: outputs: - console: enabled: yes - file: enabled: yes filename: /var/log/suricata/suricata.log - syslog: enabled: no facility: local5 format: "[%i] <%d> -- " mpipe: load-balance: dynamic iqueue-packets: 2048 inputs: - interface: xgbe2 - interface: xgbe3 - interface: xgbe4 stack: size128: 0 size256: 9 size512: 0 size1024: 0 size1664: 7 size4096: 0 size10386: 0 size16384: 0 pfring: - interface: eth0 threads: 1 cluster-id: 99 cluster-type: cluster_flow - interface: default pcap: - interface: eth0 - interface: default pcap-file: checksum-checks: auto ipfw: default-rule-path: /etc/suricata/rules rule-files: - botcc.portgrouped.rules - ciarmy.rules - compromised.rules - drop.rules - dshield.rules - emerging-activex.rules - emerging-attack_response.rules - emerging-chat.rules - emerging-current_events.rules - emerging-dns.rules - emerging-dos.rules - emerging-exploit.rules - emerging-ftp.rules - emerging-games.rules - emerging-imap.rules - emerging-inappropriate.rules - emerging-malware.rules - emerging-misc.rules - emerging-mobile_malware.rules - emerging-netbios.rules - emerging-p2p.rules - emerging-policy.rules - emerging-pop3.rules - emerging-rpc.rules - emerging-scada.rules - emerging-scan.rules - emerging-shellcode.rules - emerging-smtp.rules - emerging-snmp.rules - emerging-sql.rules - emerging-telnet.rules - emerging-tftp.rules - emerging-trojan.rules - emerging-user_agents.rules - emerging-voip.rules - emerging-web_client.rules - emerging-web_server.rules - emerging-web_specific_apps.rules - emerging-worm.rules - tor.rules - http-events.rules # available in suricata sources under rules dir - smtp-events.rules # available in suricata sources under rules dir classification-file: /etc/suricata/rules/classification.config reference-config-file: /etc/suricata/rules/reference.config vars: address-groups: HOME_NET: "[192.168.0.0/16,10.0.0.0/8,172.16.0.0/12,50.114.0.0/16,199.58.198.224/27,199.58.199.0/24,69.27.166.0/26]" EXTERNAL_NET: "!$HOME_NET" HTTP_SERVERS: "$HOME_NET" SMTP_SERVERS: "$HOME_NET" SQL_SERVERS: "$HOME_NET" DNS_SERVERS: "$HOME_NET" TELNET_SERVERS: "$HOME_NET" AIM_SERVERS: "$EXTERNAL_NET" DNP3_SERVER: "$HOME_NET" DNP3_CLIENT: "$HOME_NET" MODBUS_CLIENT: "$HOME_NET" MODBUS_SERVER: "$HOME_NET" ENIP_CLIENT: "$HOME_NET" ENIP_SERVER: "$HOME_NET" port-groups: HTTP_PORTS: "80" SHELLCODE_PORTS: "!80" ORACLE_PORTS: 1521 SSH_PORTS: 22 DNP3_PORTS: 20000 action-order: - pass - drop - reject - alert host-os-policy: windows: [] bsd: [] bsd-right: [] old-linux: [] linux: [0.0.0.0/0] old-solaris: [] solaris: [] hpux10: [] hpux11: [] irix: [] macos: [] vista: [] windows2k3: [] asn1-max-frames: 256 engine-analysis: rules-fast-pattern: yes rules: yes pcre: match-limit: 3500 match-limit-recursion: 1500 app-layer: protocols: tls: enabled: yes detection-ports: toserver: 443 dcerpc: enabled: yes ftp: enabled: yes ssh: enabled: yes smtp: enabled: yes imap: enabled: detection-only msn: enabled: detection-only smb: enabled: yes detection-ports: toserver: 139 dns: tcp: enabled: yes detection-ports: toserver: 53 udp: enabled: yes detection-ports: toserver: 53 http: enabled: yes libhtp: default-config: personality: IDS request-body-limit: 3072 response-body-limit: 3072 request-body-minimal-inspect-size: 32kb request-body-inspect-window: 4kb response-body-minimal-inspect-size: 32kb response-body-inspect-window: 4kb double-decode-path: no double-decode-query: no server-config: profiling: rules: enabled: yes filename: rule_perf.log append: yes sort: avgticks limit: 100 keywords: enabled: yes filename: keyword_perf.log append: yes packets: enabled: yes filename: packet_stats.log append: yes csv: enabled: no filename: packet_stats.csv locks: enabled: no filename: lock_stats.log append: yes coredump: max-dump: unlimited napatech: hba: -1 use-all-streams: yes streams: [1, 2, 3] </pre> Let me know if I need to provide any more information or enable features. Thanks, Jason