Project

General

Profile

Actions

Bug #705

closed

http.log missing lots of requests under high traffic load

Added by Peter Manev almost 10 years ago. Updated over 3 years ago.

Status:
Closed
Priority:
High
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

using -

 suricata --build-info
[23628] 10/1/2013 -- 00:49:59 - (suricata.c:560) <Info> (SCPrintBuildInfo) -- This is Suricata version 1.4dev (rev 5f4c528)
[23628] 10/1/2013 -- 00:49:59 - (suricata.c:633) <Info> (SCPrintBuildInfo) -- Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 PF_RING AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK HAVE_HTP_TX_GET_RESPONSE_HEADERS_RAW HAVE_NSS PROFILING
[23628] 10/1/2013 -- 00:49:59 - (suricata.c:647) <Info> (SCPrintBuildInfo) -- 64-bits, Little-endian architecture
[23628] 10/1/2013 -- 00:49:59 - (suricata.c:649) <Info> (SCPrintBuildInfo) -- GCC version 4.6.3, C version 199901
[23628] 10/1/2013 -- 00:49:59 - (suricata.c:655) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1
[23628] 10/1/2013 -- 00:49:59 - (suricata.c:658) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2
[23628] 10/1/2013 -- 00:49:59 - (suricata.c:661) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4
[23628] 10/1/2013 -- 00:49:59 - (suricata.c:664) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8
[23628] 10/1/2013 -- 00:49:59 - (suricata.c:667) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_16
[23628] 10/1/2013 -- 00:49:59 - (suricata.c:671) <Info> (SCPrintBuildInfo) -- compiled with -fstack-protector
[23628] 10/1/2013 -- 00:49:59 - (suricata.c:677) <Info> (SCPrintBuildInfo) -- compiled with _FORTIFY_SOURCE=2
[23628] 10/1/2013 -- 00:49:59 - (suricata.c:680) <Info> (SCPrintBuildInfo) -- compiled with libhtp 0.2.11, linked against 0.2.11

root@suricata:/var/data/regit/log/suricata# time tcpdump -i eth3 -n -s 0 -w - | grep -a -o -E "Host\: .*|GET \/.*"  >gets.txt
tcpdump: WARNING: eth3: no IPv4 address assigned
tcpdump: listening on eth3, link-type EN10MB (Ethernet), capture size 65535 bytes
^C637676 packets captured
4955063 packets received by filter
4317303 packets dropped by kernel
1 packet dropped by interface

real    0m3.942s
user    0m2.824s
sys     0m1.420s
root@suricata:/var/data/regit/log/suricata# grep GET gets.txt |wc -l
1529

So for 3 seconds we have 1529 HTTP GET requests with tcpdump (and that is just GET)

root@suricata:/var/data/regit/log/suricata# date && wc -l http.log
Thu Jan 10 00:14:16 EET 2013
518476 http.log
root@suricata:/var/data/regit/log/suricata# date && wc -l http.log
Thu Jan 10 00:14:18 EET 2013
518480 http.log
root@suricata:/var/data/regit/log/suricata# date && wc -l http.log
Thu Jan 10 00:14:19 EET 2013
518489 http.log


and for 3 sec with Suricata we get 13 http logs in http.log on the same 10G/s traffic
root@suricata:/var/data/regit/log/suricata# wc -l gets.txt
3209 gets.txt

root@suricata:/var/data/regit/log/suricata# tail -8 gets.txt
Host: redirector.c.youtube.com
GET /v302708503/6baa/fIDnzBb3MPE.jpg HTTP/1.1
Host: cs302708.userapi.com
GET /csi?v=2&s=youtube&action=results&e=901312,916623,922911,919108,920704,912806,922403,922405,929901,913605,929104,929110,920201,913302,919009,911116,926403,901451&li=1&rt=ct.281,js_head.591,js_page.863,js_foot.865,ol.1208,aft.1208 HTTP/1.1
Host: csi.gstatic.com
GET /socket.io/1/?t=1357769748997 HTTP/1.1
Host: ws.9gag.com:8084
GET /vi/9d6IsU01-HE/hqdefault.jpg HTTP/1.1
root@suricata:/var/data/regit/log/suricata#

my http config part in suriacta.yaml

  # a line based log of HTTP requests (no alerts)
  - http-log:
      enabled: yes
      filename: http.log
      append: yes
      extended: yes     # enable this for extended logging information
      #filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'

just for general info - extended: yes/no - it has no effect on the number of logs written.

I think 1.3.1 didn't have that issue.

Actions

Also available in: Atom PDF