Project

General

Profile

Actions
Copy link

Bug #1637

closed

drop log crashes

Added by Hayder Sinan almost 10 years ago. Updated over 9 years ago.

Status:
Closed
Priority:
High
Assignee:
Victor Julien
Target version:
2.0.11
Affected Versions:
Effort:
Difficulty:
Label:

Description

I have a Suricata running on version 2.0.10 on Centos7 we have an issue that the Suricata service fails when it receives a small udp packet size or a large one please find below logs when the issue happens

the following logs are taken when it received a large udp Packet Size

[root@ips01 suricata]# tail /var/log/messages
Dec  8 14:23:59 ips01 iptables.init: iptables: Setting chains to policy ACCEPT: mangle filter [  OK  ]
Dec  8 14:23:59 ips01 iptables.init: iptables: Flushing firewall rules: [  OK  ]
Dec  8 14:23:59 ips01 iptables.init: iptables: Unloading modules: [  OK  ]
Dec  8 14:23:59 ips01 systemd: Starting IPv4 firewall with iptables...
Dec  8 14:23:59 ips01 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Dec  8 14:23:59 ips01 iptables.init: iptables: Applying firewall rules: [  OK  ]
Dec  8 14:23:59 ips01 systemd: Started IPv4 firewall with iptables.
Dec  8 14:24:09 ips01 kernel: Detect1[26508]: segfault at 4 ip 00000000004c7fcc sp 00007f4fac0a0470 error 4 in suricata[400000+1c6000]
Dec  8 14:24:09 ips01 systemd: suricata.service: main process exited, code=killed, status=11/SEGV
Dec  8 14:24:09 ips01 systemd: Unit suricata.service entered failed state.

[root@ips01 ~]# tailf /usr/local/var/log/suricata/fast.log
12/08/2015-14:24:07.437108  [Drop] [**] [1:521:2] MISC Large UDP Packet [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.236.200.100:53 -> 194.18.169.45:26370
12/08/2015-14:24:07.438062  [Drop] [**] [1:521:2] MISC Large UDP Packet [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.236.200.100:53 -> 194.18.169.45:26370
12/08/2015-14:24:07.441106  [Drop] [**] [1:521:2] MISC Large UDP Packet [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.236.200.100:53 -> 194.18.169.45:26370
12/08/2015-14:24:07.871481  [Drop] [**] [1:2017645:2] ET CURRENT_EVENTS DNS Query Domain .bit [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.239.62.144:60723 -> 87.98.175.85:53
12/08/2015-14:24:08.235181  [Drop] [**] [1:2017645:2] ET CURRENT_EVENTS DNS Query Domain .bit [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.239.62.144:63715 -> 106.186.17.181:53
12/08/2015-14:24:08.344190  [Drop] [**] [1:2017645:2] ET CURRENT_EVENTS DNS Query Domain .bit [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.239.62.144:63328 -> 95.211.195.245:53
12/08/2015-14:24:08.466187  [Drop] [**] [1:2017645:2] ET CURRENT_EVENTS DNS Query Domain .bit [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.239.62.144:57532 -> 192.121.170.170:53
12/08/2015-14:24:08.653227  [Drop] [**] [1:2017645:2] ET CURRENT_EVENTS DNS Query Domain .bit [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.239.62.144:60581 -> 151.236.6.6:53
12/08/2015-14:24:08.887174  [Drop] [**] [1:2017645:2] ET CURRENT_EVENTS DNS Query Domain .bit [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.239.62.144:60723 -> 87.98.175.85:53
12/08/2015-14:24:09.075701  [Drop] [**] [1:521:2] MISC Large UDP Packet [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.239.196.31:53 -> 85.195.64.11:27710

if we disable the Sid #521 or sid #2200038 the issue is solved and the service doesn't fail can you please help?

the rules are 
drop pkthdr any any -> any any (msg:"SURICATA UDP packet too small"; decode-event:udp.pkt_too_small; sid:2200038; rev:1;)

drop udp any any -> any any (msg:"MISC Large UDP Packet"; dsize:>4000; reference:arachnids,247; classtype:bad-unknown; sid:521; rev:2;)

Files

Download all files
gdb-Core-12720.txt (3.83 KB) gdb-Core-12720.txt Hayder Sinan, 12/09/2015 04:27 AM
gdb-Core-26557.txt (1.75 KB) gdb-Core-26557.txt Hayder Sinan, 12/09/2015 04:27 AM
IPS.txt (2.57 KB) IPS.txt suricata --build-info Hayder Sinan, 12/09/2015 04:29 AM
Actions
Copy link
 #1

Updated by Victor Julien almost 10 years ago

Can you record the traffic and attach the pcap?

Actions
Copy link
 #2

Updated by Victor Julien almost 10 years ago

  • Description updated (diff)
Actions
Copy link
 #3

Updated by Hayder Sinan almost 10 years ago

Dear Victor

please find below Google Drive Link for the pcap file please note that the capture was taken while the rules where commented if you like I can repeat it while it's uncommented

https://drive.google.com/file/d/0B8PgMIlwY8MLdU90aWctZVBpNW8/view?usp=sharing

Actions
Copy link
 #4

Updated by Hayder Sinan almost 10 years ago

Dear,

please find below logs it happened when the it received a small udp packet

[root@ips01 ~]# tail -n 40 /usr/local/var/log/suricata/fast.log
12/08/2015-17:12:51.768280 [Drop] [**] [1:2014703:8] ET DNS Non-DNS or Non-Compliant DNS traffic on DNS port Reserved Bit Set - Likely Kazy [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {UDP} 37.239.0.47:14762 -> 106.38.187.142:53
12/08/2015-17:12:52.449464 [Drop] [**] [1:2012811:2] ET DNS DNS Query to a .tk domain - Likely Hostile [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.237.83.33:49409 -> 199.203.131.145:53
12/08/2015-17:12:53.200179 [Drop] [**] [1:2012811:2] ET DNS DNS Query to a .tk domain - Likely Hostile [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.237.83.33:58325 -> 199.203.131.145:53
12/08/2015-17:12:53.924091 [Drop] [**] [1:2009702:5] ET POLICY DNS Update From External net [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {UDP} 37.239.255.254:55677 -> 67.23.224.151:53
12/08/2015-17:12:54.557045 [Drop] [**] [1:2012811:2] ET DNS DNS Query to a .tk domain - Likely Hostile [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.237.83.33:49409 -> 199.203.131.145:53
12/08/2015-17:12:56.643216 [Drop] [**] [1:2014703:8] ET DNS Non-DNS or Non-Compliant DNS traffic on DNS port Reserved Bit Set - Likely Kazy [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {UDP} 37.239.0.6:12620 -> 168.1.69.109:53
12/08/2015-17:12:57.843547 [Drop] [**] [1:2014169:1] ET POLICY DNS Query for .su TLD (Soviet Union) Often Malware Related [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.239.0.29:11268 -> 8.8.4.4:53
12/08/2015-17:12:58.041774 [Drop] [**] [1:2009702:5] ET POLICY DNS Update From External net [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {UDP} 37.236.200.93:11232 -> 192.131.182.241:53
12/08/2015-17:13:07.651615 [Drop] [**] [1:2012811:2] ET DNS DNS Query to a .tk domain - Likely Hostile [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} 37.237.83.33:29469 -> 199.203.131.145:53
12/08/2015-17:13:14.475466 [Drop] [**] [1:2009702:5] ET POLICY DNS Update From External net [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {UDP} 37.239.255.254:53927 -> 67.23.224.151:53
12/08/2015-17:13:18.152123 [Drop] [**] [1:2014703:8] ET DNS Non-DNS or Non-Compliant DNS traffic on DNS port Reserved Bit Set - Likely Kazy [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {UDP} 37.238.212.154:64540 -> 54.76.137.57:53
12/08/2015-17:13:19.417191 [Drop] [**] [1:2009702:5] ET POLICY DNS Update From External net [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {UDP} 37.236.200.93:8872 -> 192.131.182.241:53
12/08/2015-17:13:19.474715 [Drop] [**] [1:2009702:5] ET POLICY DNS Update From External net [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {UDP} 37.239.255.254:53927 -> 67.23.224.151:53
12/08/2015-17:13:23.991153 [Drop] [**] [1:2200038:1] SURICATA UDP packet too small [**] [Classification: (null)] [Priority: 3] {UDP} 37.236.200.100:0 -> 72.220.237.17:0

[root@ips01 ~]# tail /var/log/messages -n 20
Dec 8 15:13:24 ips01 systemd: Started Session 92 of user root.
Dec 8 15:13:44 ips01 kernel: device eno1 left promiscuous mode
Dec 8 16:01:01 ips01 systemd: Starting Session 93 of user root.
Dec 8 16:01:01 ips01 systemd: Started Session 93 of user root.
Dec 8 17:01:01 ips01 systemd: Starting Session 94 of user root.
Dec 8 17:01:01 ips01 systemd: Started Session 94 of user root.
Dec 8 17:13:23 ips01 kernel: Detect1226653: segfault at 4 ip 00000000004c7fcc sp 00007fa7c67fb470 error 4 in suricata[400000+1c6000]
Dec 8 17:13:24 ips01 systemd: suricata.service: main process exited, code=killed, status=11/SEGV
Dec 8 17:13:24 ips01 systemd: Unit suricata.service entered failed state.

Actions
Copy link
 #5

Updated by Victor Julien almost 10 years ago

  • Priority changed from High to Normal

Using just the pcap I see no issues. Can you try to get a backtrace from the crash? See https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Reporting_Bugs

Actions
Copy link Download all files
 #6

Updated by Hayder Sinan almost 10 years ago

Dear ,

there were multiple Core Files I have done the GDP process for two of them if you want I can upload the entire Core files to Google drive so that you check them
Actions
Copy link
 #7

Updated by Victor Julien almost 10 years ago

Sadly, these bt's contain too little info. We need a binary w/o stripped symbols.

Can you recompile Suricata with CFLAGS="-ggdb", reproduce the issue and create another backtrace?

Actions
Copy link
 #8

Updated by Hayder Sinan almost 10 years ago

Dear,

we have already rebuild Suricata with the following flag CFLAGS="-ggdb -O0" showed we change it to CFLAGS="-ggdb" this the command we used for the rebuild

./configure --enable-nfqueue CFLAGS="-ggdb -O0" && make && make install-full

Actions
Copy link
 #9

Updated by Victor Julien almost 10 years ago

Something else must have gone wrong then, as the bt's don't show the symbols and are therefore not usable.

Actions
Copy link
 #10

Updated by Hayder Sinan almost 10 years ago

ok then we will try to rebuild the Suricata again with the following flag CFLAGS="-ggdb" is that ok and is there a way to check if the build has enabled the ggdb flag as stated in the command?

Actions
Copy link
 #11

Updated by Victor Julien almost 10 years ago

During the "make" stage, each gcc line should show this -ggdb option. Be sure to make clean first.

Actions
Copy link
 #12

Updated by Hayder Sinan almost 10 years ago

Dear 

I have done make clean and install it again and there a -ggdb in each ggc line and then I have done the following three steps
gdb /usr/local/bin/suricata /core
(gdb) set logging on
(gdb) thread apply all bt
it created the exact same txt file that I have send you is there any recommended steps that I can follow?
Actions
Copy link
 #13

Updated by Victor Julien almost 10 years ago

  • Status changed from New to Assigned
  • Assignee set to Victor Julien
  • Priority changed from Normal to High
  • Target version set to 2.0.11

I have been able to reproduce this. As a work around, you can disable the drop log output or the eve drop log. Will release a fix soon.

Actions
Copy link
 #14

Updated by Hayder Sinan almost 10 years ago

Dear,

Thanks for your help the work around has worked and the system is working greatly
Actions
Copy link
 #15

Updated by Victor Julien over 9 years ago

  • Subject changed from Suricata Fails if receives a udp small or Large size packet to drop log crashes
Actions
Copy link
 #16

Updated by Victor Julien over 9 years ago

  • Status changed from Assigned to Closed
  • % Done changed from 0 to 100
Actions
Copy link
 #17

Updated by Hayder Sinan over 9 years ago

Thanks For Your Help we have successfully updated our system and it's working normally

Actions
Copy link

Also available in: Atom PDF