Bug #254
closedhttp.log doesn't log all the Http Get Requests
Description
It seems that http.log doesn't log all the Http Get Requests. At least when compared to a tcpdump or tshark pcap file.
Most of the requests that http.log is missing are:
*.jpg
*.js
*.css
*.png
**.swf
Please find attached a small script that I wrote to compare a pcap with http.log. Please make sure you have tshark installed.
basic usage is :
./tcomppcaphttplog pcap_file http.log result
You can open up a couple of web pages to compare the different outputs.
Files
Updated by Victor Julien almost 14 years ago
Can you attach a pcap with reproducible missing requests?
Updated by Peter Manev over 13 years ago
- File http.log http.log added
- File tsharkdump tsharkdump added
- File unmatchedhttp.log unmatchedhttp.log added
- File unmatchedhttpgettsharkdump unmatchedhttpgettsharkdump added
- File result result added
Compiled and installed Suricata from git.
Started Suricata , waited /about 30 sec/ so it starts logging info in the "stats1.log".
Then I started the thsark dump, then visited the following websites:
www.cisco.com , www.youtube.com , www.cnn.com , www.facebook.com .
Please find attached http.log and tsharkdump pcap files.
I used the script (tcomppcaphttplog) previously uploaded to check for differences.
MOST of the http_get requests that were unmatched from the http.log in the tshark pcap file are actually there (in the tshark pcap), just the tshark pcap is missing the last letter/symbol.
However there are some http_get requests that were not found in the http.log.
Thanks
Updated by Victor Julien over 13 years ago
- Status changed from New to Assigned
- Assignee set to Victor Julien
It appears that all missing requests from the http.log can be attributed to the following:
1. many of the connections are not closed properly: the entire FIN/RST sequence is missing.
2. some of the connections have missing packets causing the http parser to give up.
For (1) the reason we're not printing the request uri's is that currently we only consider a transaction ready for printing when the full transaction is received. For the final transaction in a connection we rely on the proper connection shutdown (either through FIN or RST) to reach this state.
The solution for this would be to keep separate track of requests that are ready for logging, despite the response part of the same transaction not being ready yet. For this we could hook into the "request done" callback.
Updated by Victor Julien almost 13 years ago
- Assignee changed from Victor Julien to Peter Manev
- Estimated time set to 3.00 h
Peter, can you revisit this with the git master? It should be way more accurate.
Updated by Peter Manev almost 13 years ago
- % Done changed from 0 to 100
Checked
Suricata current git
Suricata Beta 3 rev 18da4a8
Suricata 1.0.6
Judging from the results we match 100% all http requests.
Updated by Victor Julien almost 13 years ago
- Status changed from Assigned to Closed
- Target version set to 1.1beta3
Thanks Peter.