Project

General

Profile

Actions

Bug #254

closed

http.log doesn't log all the Http Get Requests

Added by Peter Manev almost 11 years ago. Updated almost 10 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

It seems that http.log doesn't log all the Http Get Requests. At least when compared to a tcpdump or tshark pcap file.
Most of the requests that http.log is missing are:
*.jpg
*.js
*.css
*.png
**.swf

Please find attached a small script that I wrote to compare a pcap with http.log. Please make sure you have tshark installed.
basic usage is :
./tcomppcaphttplog pcap_file http.log result
You can open up a couple of web pages to compare the different outputs.


Files

tcomppcaphttplog (4.5 KB) tcomppcaphttplog Peter Manev, 12/16/2010 07:25 AM
http.log (59 KB) http.log http.log from Suricata Peter Manev, 01/06/2011 03:19 AM
tsharkdump (4.2 MB) tsharkdump pcap file generated using tshark Peter Manev, 01/06/2011 03:19 AM
unmatchedhttp.log (1.44 KB) unmatchedhttp.log unmatched http_get requests in the http.log file - not found in the tshark pcap Peter Manev, 01/06/2011 03:19 AM
unmatchedhttpgettsharkdump (4.29 KB) unmatchedhttpgettsharkdump unmatched http_get requests in the tshark pcap file - not found in the http.log Peter Manev, 01/06/2011 03:19 AM
result (124 KB) result matched http_get requests - found in both tshark pcap and http.log Peter Manev, 01/06/2011 03:19 AM
Actions #1

Updated by Victor Julien almost 11 years ago

Can you attach a pcap with reproducible missing requests?

Actions #2

Updated by Peter Manev almost 11 years ago

Compiled and installed Suricata from git.
Started Suricata , waited /about 30 sec/ so it starts logging info in the "stats1.log".
Then I started the thsark dump, then visited the following websites:
www.cisco.com , www.youtube.com , www.cnn.com , www.facebook.com .

Please find attached http.log and tsharkdump pcap files.
I used the script (tcomppcaphttplog) previously uploaded to check for differences.

MOST of the http_get requests that were unmatched from the http.log in the tshark pcap file are actually there (in the tshark pcap), just the tshark pcap is missing the last letter/symbol.
However there are some http_get requests that were not found in the http.log.

Thanks

Actions #3

Updated by Victor Julien over 10 years ago

  • Status changed from New to Assigned
  • Assignee set to Victor Julien

It appears that all missing requests from the http.log can be attributed to the following:

1. many of the connections are not closed properly: the entire FIN/RST sequence is missing.

2. some of the connections have missing packets causing the http parser to give up.

For (1) the reason we're not printing the request uri's is that currently we only consider a transaction ready for printing when the full transaction is received. For the final transaction in a connection we rely on the proper connection shutdown (either through FIN or RST) to reach this state.

The solution for this would be to keep separate track of requests that are ready for logging, despite the response part of the same transaction not being ready yet. For this we could hook into the "request done" callback.

Actions #4

Updated by Victor Julien about 10 years ago

  • Assignee changed from Victor Julien to Peter Manev
  • Estimated time set to 3.00 h

Peter, can you revisit this with the git master? It should be way more accurate.

Actions #5

Updated by Peter Manev almost 10 years ago

  • % Done changed from 0 to 100

Checked
Suricata current git
Suricata Beta 3 rev 18da4a8
Suricata 1.0.6

Judging from the results we match 100% all http requests.

Actions #6

Updated by Victor Julien almost 10 years ago

  • Status changed from Assigned to Closed
  • Target version set to 1.1beta3

Thanks Peter.

Actions

Also available in: Atom PDF