Project

General

Profile

Actions
Copy link

Bug #1637

closed

drop log crashes

Added by Hayder Sinan almost 9 years ago. Updated almost 9 years ago.

Status:
Closed
Priority:
High
Assignee:
Victor Julien
Target version:
2.0.11
Affected Versions:
Effort:
Difficulty:
Label:

Description

I have a Suricata running on version 2.0.10 on Centos7 we have an issue that the Suricata service fails when it receives a small udp packet size or a large one please find below logs when the issue happens

the following logs are taken when it received a large udp Packet Size

[root@ips01 suricata]# tail /var/log/messages
Dec  8 14:23:59 ips01 iptables.init: iptables: Setting chains to policy ACCEPT: mangle filter [  OK  ]
Dec  8 14:23:59 ips01 iptables.init: iptables: Flushing firewall rules: [  OK  ]
Dec  8 14:23:59 ips01 iptables.init: iptables: Unloading modules: [  OK  ]
Dec  8 14:23:59 ips01 systemd: Starting IPv4 firewall with iptables...
Dec  8 14:23:59 ips01 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Dec  8 14:23:59 ips01 iptables.init: iptables: Applying firewall rules: [  OK  ]
Dec  8 14:23:59 ips01 systemd: Started IPv4 firewall with iptables.
Dec  8 14:24:09 ips01 kernel: Detect1[26508]: segfault at 4 ip 00000000004c7fcc sp 00007f4fac0a0470 error 4 in suricata[400000+1c6000]
Dec  8 14:24:09 ips01 systemd: suricata.service: main process exited, code=killed, status=11/SEGV
Dec  8 14:24:09 ips01 systemd: Unit suricata.service entered failed state.

[root@ips01 ~]# tailf /usr/local/var/log/suricata/fast.log
12/08/2015-14:24:07.437108  [Drop] [**] [1:521:2] MISC Large UDP Packet [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.236.200.100:53 -> 194.18.169.45:26370
12/08/2015-14:24:07.438062  [Drop] [**] [1:521:2] MISC Large UDP Packet [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.236.200.100:53 -> 194.18.169.45:26370
12/08/2015-14:24:07.441106  [Drop] [**] [1:521:2] MISC Large UDP Packet [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.236.200.100:53 -> 194.18.169.45:26370
12/08/2015-14:24:07.871481  [Drop] [**] [1:2017645:2] ET CURRENT_EVENTS DNS Query Domain .bit [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.239.62.144:60723 -> 87.98.175.85:53
12/08/2015-14:24:08.235181  [Drop] [**] [1:2017645:2] ET CURRENT_EVENTS DNS Query Domain .bit [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.239.62.144:63715 -> 106.186.17.181:53
12/08/2015-14:24:08.344190  [Drop] [**] [1:2017645:2] ET CURRENT_EVENTS DNS Query Domain .bit [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.239.62.144:63328 -> 95.211.195.245:53
12/08/2015-14:24:08.466187  [Drop] [**] [1:2017645:2] ET CURRENT_EVENTS DNS Query Domain .bit [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.239.62.144:57532 -> 192.121.170.170:53
12/08/2015-14:24:08.653227  [Drop] [**] [1:2017645:2] ET CURRENT_EVENTS DNS Query Domain .bit [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.239.62.144:60581 -> 151.236.6.6:53
12/08/2015-14:24:08.887174  [Drop] [**] [1:2017645:2] ET CURRENT_EVENTS DNS Query Domain .bit [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.239.62.144:60723 -> 87.98.175.85:53
12/08/2015-14:24:09.075701  [Drop] [**] [1:521:2] MISC Large UDP Packet [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.239.196.31:53 -> 85.195.64.11:27710

if we disable the Sid #521 or sid #2200038 the issue is solved and the service doesn't fail can you please help?

the rules are 
drop pkthdr any any -> any any (msg:"SURICATA UDP packet too small"; decode-event:udp.pkt_too_small; sid:2200038; rev:1;)

drop udp any any -> any any (msg:"MISC Large UDP Packet"; dsize:>4000; reference:arachnids,247; classtype:bad-unknown; sid:521; rev:2;)

Files

Download all files
gdb-Core-12720.txt (3.83 KB) gdb-Core-12720.txt Hayder Sinan, 12/09/2015 04:27 AM
gdb-Core-26557.txt (1.75 KB) gdb-Core-26557.txt Hayder Sinan, 12/09/2015 04:27 AM
IPS.txt (2.57 KB) IPS.txt suricata --build-info Hayder Sinan, 12/09/2015 04:29 AM
Actions
Copy link
 #1

Updated by Victor Julien almost 9 years ago

Can you record the traffic and attach the pcap?

Actions
Copy link
 #2

Updated by Victor Julien almost 9 years ago

  • Description updated (diff)
Actions
Copy link
 #3

Updated by Hayder Sinan almost 9 years ago

Dear Victor

please find below Google Drive Link for the pcap file please note that the capture was taken while the rules where commented if you like I can repeat it while it's uncommented

https://drive.google.com/file/d/0B8PgMIlwY8MLdU90aWctZVBpNW8/view?usp=sharing

Actions
Copy link
 #4

Updated by Hayder Sinan almost 9 years ago

Dear,

please find below logs it happened when the it received a small udp packet

[root@ips01 ~]# tail -n 40 /usr/local/var/log/suricata/fast.log
12/08/2015-17:12:51.768280 [Drop] [**] [1:2014703:8] ET DNS Non-DNS or Non-Compliant DNS traffic on DNS port Reserved Bit Set - Likely Kazy [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {UDP} 37.239.0.47:14762 -> 106.38.187.142:53
12/08/2015-17:12:52.449464 [Drop] [**] [1:2012811:2] ET DNS DNS Query to a .tk domain - Likely Hostile [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.237.83.33:49409 -> 199.203.131.145:53
12/08/2015-17:12:53.200179 [Drop] [**] [1:2012811:2] ET DNS DNS Query to a .tk domain - Likely Hostile [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.237.83.33:58325 -> 199.203.131.145:53
12/08/2015-17:12:53.924091 [Drop] [**] [1:2009702:5] ET POLICY DNS Update From External net [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {UDP} 37.239.255.254:55677 -> 67.23.224.151:53
12/08/2015-17:12:54.557045 [Drop] [**] [1:2012811:2] ET DNS DNS Query to a .tk domain - Likely Hostile [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.237.83.33:49409 -> 199.203.131.145:53
12/08/2015-17:12:56.643216 [Drop] [**] [1:2014703:8] ET DNS Non-DNS or Non-Compliant DNS traffic on DNS port Reserved Bit Set - Likely Kazy [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {UDP} 37.239.0.6:12620 -> 168.1.69.109:53
12/08/2015-17:12:57.843547 [Drop] [**] [1:2014169:1] ET POLICY DNS Query for .su TLD (Soviet Union) Often Malware Related [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.239.0.29:11268 -> 8.8.4.4:53
12/08/2015-17:12:58.041774 [Drop] [**] [1:2009702:5] ET POLICY DNS Update From External net [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {UDP} 37.236.200.93:11232 -> 192.131.182.241:53
12/08/2015-17:13:07.651615 [Drop] [**] [1:2012811:2] ET DNS DNS Query to a .tk domain - Likely Hostile [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.237.83.33:29469 -> 199.203.131.145:53
12/08/2015-17:13:14.475466 [Drop] [**] [1:2009702:5] ET POLICY DNS Update From External net [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {UDP} 37.239.255.254:53927 -> 67.23.224.151:53
12/08/2015-17:13:18.152123 [Drop] [**] [1:2014703:8] ET DNS Non-DNS or Non-Compliant DNS traffic on DNS port Reserved Bit Set - Likely Kazy [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {UDP} 37.238.212.154:64540 -> 54.76.137.57:53
12/08/2015-17:13:19.417191 [Drop] [**] [1:2009702:5] ET POLICY DNS Update From External net [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {UDP} 37.236.200.93:8872 -> 192.131.182.241:53
12/08/2015-17:13:19.474715 [Drop] [**] [1:2009702:5] ET POLICY DNS Update From External net [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {UDP} 37.239.255.254:53927 -> 67.23.224.151:53
12/08/2015-17:13:23.991153 [Drop] [**] [1:2200038:1] SURICATA UDP packet too small [**] [Classification: (null)] [Priority: 3] {UDP} 37.236.200.100:0 -> 72.220.237.17:0

[root@ips01 ~]# tail /var/log/messages -n 20
Dec 8 15:13:24 ips01 systemd: Started Session 92 of user root.
Dec 8 15:13:44 ips01 kernel: device eno1 left promiscuous mode
Dec 8 16:01:01 ips01 systemd: Starting Session 93 of user root.
Dec 8 16:01:01 ips01 systemd: Started Session 93 of user root.
Dec 8 17:01:01 ips01 systemd: Starting Session 94 of user root.
Dec 8 17:01:01 ips01 systemd: Started Session 94 of user root.
Dec 8 17:13:23 ips01 kernel: Detect1226653: segfault at 4 ip 00000000004c7fcc sp 00007fa7c67fb470 error 4 in suricata[400000+1c6000]
Dec 8 17:13:24 ips01 systemd: suricata.service: main process exited, code=killed, status=11/SEGV
Dec 8 17:13:24 ips01 systemd: Unit suricata.service entered failed state.

Actions
Copy link
 #5

Updated by Victor Julien almost 9 years ago

  • Priority changed from High to Normal

Using just the pcap I see no issues. Can you try to get a backtrace from the crash? See https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Reporting_Bugs

Actions
Copy link Download all files
 #6

Updated by Hayder Sinan almost 9 years ago

Dear ,

there were multiple Core Files I have done the GDP process for two of them if you want I can upload the entire Core files to Google drive so that you check them
Actions
Copy link
 #7

Updated by Victor Julien almost 9 years ago

Sadly, these bt's contain too little info. We need a binary w/o stripped symbols.

Can you recompile Suricata with CFLAGS="-ggdb", reproduce the issue and create another backtrace?

Actions
Copy link
 #8

Updated by Hayder Sinan almost 9 years ago

Dear,

we have already rebuild Suricata with the following flag CFLAGS="-ggdb -O0" showed we change it to CFLAGS="-ggdb" this the command we used for the rebuild

./configure --enable-nfqueue CFLAGS="-ggdb -O0" && make && make install-full

Actions
Copy link
 #9

Updated by Victor Julien almost 9 years ago

Something else must have gone wrong then, as the bt's don't show the symbols and are therefore not usable.

Actions
Copy link
 #10

Updated by Hayder Sinan almost 9 years ago

ok then we will try to rebuild the Suricata again with the following flag CFLAGS="-ggdb" is that ok and is there a way to check if the build has enabled the ggdb flag as stated in the command?

Actions
Copy link
 #11

Updated by Victor Julien almost 9 years ago

During the "make" stage, each gcc line should show this -ggdb option. Be sure to make clean first.

Actions
Copy link
 #12

Updated by Hayder Sinan almost 9 years ago

Dear 

I have done make clean and install it again and there a -ggdb in each ggc line and then I have done the following three steps
gdb /usr/local/bin/suricata /core
(gdb) set logging on
(gdb) thread apply all bt
it created the exact same txt file that I have send you is there any recommended steps that I can follow?
Actions
Copy link
 #13

Updated by Victor Julien almost 9 years ago

  • Status changed from New to Assigned
  • Assignee set to Victor Julien
  • Priority changed from Normal to High
  • Target version set to 2.0.11

I have been able to reproduce this. As a work around, you can disable the drop log output or the eve drop log. Will release a fix soon.

Actions
Copy link
 #14

Updated by Hayder Sinan almost 9 years ago

Dear,

Thanks for your help the work around has worked and the system is working greatly
Actions
Copy link
 #15

Updated by Victor Julien almost 9 years ago

  • Subject changed from Suricata Fails if receives a udp small or Large size packet to drop log crashes
Actions
Copy link
 #16

Updated by Victor Julien almost 9 years ago

  • Status changed from Assigned to Closed
  • % Done changed from 0 to 100
Actions
Copy link
 #17

Updated by Hayder Sinan almost 9 years ago

Thanks For Your Help we have successfully updated our system and it's working normally

Actions
Copy link

Also available in: Atom PDF