http.log doesn't log all the Http Get Requests
It seems that http.log doesn't log all the Http Get Requests. At least when compared to a tcpdump or tshark pcap file.
Most of the requests that http.log is missing are:
Please find attached a small script that I wrote to compare a pcap with http.log. Please make sure you have tshark installed.
basic usage is :
./tcomppcaphttplog pcap_file http.log result
You can open up a couple of web pages to compare the different outputs.
Updated by Peter Manev over 11 years ago
- File http.log http.log added
- File tsharkdump tsharkdump added
- File unmatchedhttp.log unmatchedhttp.log added
- File unmatchedhttpgettsharkdump unmatchedhttpgettsharkdump added
- File result result added
Compiled and installed Suricata from git.
Started Suricata , waited /about 30 sec/ so it starts logging info in the "stats1.log".
Then I started the thsark dump, then visited the following websites:
www.cisco.com , www.youtube.com , www.cnn.com , www.facebook.com .
Please find attached http.log and tsharkdump pcap files.
I used the script (tcomppcaphttplog) previously uploaded to check for differences.
MOST of the http_get requests that were unmatched from the http.log in the tshark pcap file are actually there (in the tshark pcap), just the tshark pcap is missing the last letter/symbol.
However there are some http_get requests that were not found in the http.log.
Updated by Victor Julien over 11 years ago
- Status changed from New to Assigned
- Assignee set to Victor Julien
It appears that all missing requests from the http.log can be attributed to the following:
1. many of the connections are not closed properly: the entire FIN/RST sequence is missing.
2. some of the connections have missing packets causing the http parser to give up.
For (1) the reason we're not printing the request uri's is that currently we only consider a transaction ready for printing when the full transaction is received. For the final transaction in a connection we rely on the proper connection shutdown (either through FIN or RST) to reach this state.
The solution for this would be to keep separate track of requests that are ready for logging, despite the response part of the same transaction not being ready yet. For this we could hook into the "request done" callback.