Bug #254: http.log doesn't log all the Http Get Requests - Suricata - Open Information Security Foundation

Actions

Copy link

Bug #254

closed

http.log doesn't log all the Http Get Requests

Added by Peter Manev about 14 years ago. Updated about 13 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Peter Manev

Target version:

1.1beta3

Affected Versions:

Effort:

Difficulty:

Label:

Description

It seems that http.log doesn't log all the Http Get Requests. At least when compared to a tcpdump or tshark pcap file.
Most of the requests that http.log is missing are:
*.jpg
*.js
*.css
*.png
**.swf

Please find attached a small script that I wrote to compare a pcap with http.log. Please make sure you have tshark installed.
basic usage is :
./tcomppcaphttplog pcap_file http.log result
You can open up a couple of web pages to compare the different outputs.

Files

Download all files

tcomppcaphttplog (4.5 KB) tcomppcaphttplog		Peter Manev, 12/16/2010 07:25 AM
http.log (59 KB) http.log	http.log from Suricata	Peter Manev, 01/06/2011 03:19 AM
tsharkdump (4.2 MB) tsharkdump	pcap file generated using tshark	Peter Manev, 01/06/2011 03:19 AM
unmatchedhttp.log (1.44 KB) unmatchedhttp.log	unmatched http_get requests in the http.log file - not found in the tshark pcap	Peter Manev, 01/06/2011 03:19 AM
unmatchedhttpgettsharkdump (4.29 KB) unmatchedhttpgettsharkdump	unmatched http_get requests in the tshark pcap file - not found in the http.log	Peter Manev, 01/06/2011 03:19 AM
result (124 KB) result	matched http_get requests - found in both tshark pcap and http.log	Peter Manev, 01/06/2011 03:19 AM

Actions

Copy link

Updated by Victor Julien almost 14 years ago

Can you attach a pcap with reproducible missing requests?

Actions

Copy link Download all files

Updated by Peter Manev almost 14 years ago

File http.log http.log added
File tsharkdump tsharkdump added
File unmatchedhttp.log unmatchedhttp.log added
File unmatchedhttpgettsharkdump unmatchedhttpgettsharkdump added
File result result added

Compiled and installed Suricata from git.
Started Suricata , waited /about 30 sec/ so it starts logging info in the "stats1.log".
Then I started the thsark dump, then visited the following websites:
www.cisco.com , www.youtube.com , www.cnn.com , www.facebook.com .

Please find attached http.log and tsharkdump pcap files.
I used the script (tcomppcaphttplog) previously uploaded to check for differences.

MOST of the http_get requests that were unmatched from the http.log in the tshark pcap file are actually there (in the tshark pcap), just the tshark pcap is missing the last letter/symbol.
However there are some http_get requests that were not found in the http.log.

Thanks

Actions

Copy link

Updated by Victor Julien almost 14 years ago

Status changed from New to Assigned
Assignee set to Victor Julien

It appears that all missing requests from the http.log can be attributed to the following:

1. many of the connections are not closed properly: the entire FIN/RST sequence is missing.

2. some of the connections have missing packets causing the http parser to give up.

For (1) the reason we're not printing the request uri's is that currently we only consider a transaction ready for printing when the full transaction is received. For the final transaction in a connection we rely on the proper connection shutdown (either through FIN or RST) to reach this state.

The solution for this would be to keep separate track of requests that are ready for logging, despite the response part of the same transaction not being ready yet. For this we could hook into the "request done" callback.

Actions

Copy link