Feature #478
closedXFF (X-Forwarded-For)
Description
XFF support so that Suricata could be placed behind a reverse proxy which terminates the SSL and SNATs the HTTP connections to the web servers.
Currently, in this scenario Suricata sees all inbound HTTP connections as coming from the internal IP address of the reverse proxy. Therefore if an attack is detected, the reverse proxy IP address will be recorded as the attacker's IP. The http-log will also just display the reverse proxy IP.
Reverse proxies, when deployed in non transparent mode, typically insert the original client IP address into an HTTP header such as X-Forwarded-For.
Snort, since version 2.9.0 supports the extraction of this header (--enable_xff) and inserts it as a extrahdr record in the unified2 files.
Suggested features:
- Insert into an extrahdr unified2 record the actual client IP address if it is present in the HTTP header of the HTTP request which triggered the alert. XFF feature and XFF header name could be configured via suricata's yaml configuration file.
- Add client XFF IP address to the extended http-log (if XFF is configured via suricata's yaml configuration file).
Files
Updated by Victor Julien over 12 years ago
- Target version set to TBD
Adding the xff header to the http.log will be trivial I think. Chris, you interested in taking that on?
Wrt Unified2, we probably first need to get our u2 updated. We use a slightly older file format currently.
Updated by Ignacio Sanchez about 12 years ago
Ok. I attach a quick patch I have developed to add the X-Forwarded-For header, if present, to the extended http log.
I am considering adding a feature to allow the user specify a custom http log format via the yaml configuration file. This way, the user could customize the order, separator and the HTTP fields to be included in the http log.
What do you think?
Updated by Victor Julien about 12 years ago
Yeah this looks good. Might be interesting to add it to the file extraction/inspection related outputs as well, such as the files-json.log and the file store meta files.
Updated by Ignacio Sanchez about 12 years ago
- File 0001-Custom-logging-feature-for-log-httplog.patch added
- File log-httplog.c added
Ok, I have created the following patch for the log-httplog which adds support for custom http logging using a format syntax inspired by Apache mod_log_config.
In order to activate the custom logging feature, the parameters custom and customformat shall be specified in the suricata.yaml configuration file.
Example (next to "extended" under http-log:
custom: yes # enable the custom logging format (defined by customformat)
customformat: "%{%D-%H:%M:%S}t.%z %{X-Forwarded-For}i %H %m %h %u %s %B %a:%p -> %A:%P"
In addition to %h, %H, %m, %u, %i, %C, %s, %o and %B - almost - as described by mod_log_config (http://httpd.apache.org/docs/2.0/mod/mod_log_config.html), I have added %z, %a, %p, %A and %P for the precision time, IPs and ports.
I have tested it in suricata 1.3.1b2 and in the latest suricata git version at the time of writing and it seems to be working fine.
As illustrated by the example, the XFF client IP can be logged with "%{X-Forwarded-For}i" and using the right customformat string the HTTP transaction log files would be directly readable by awstats or piwik so now we can have real time statistics of the monitored web applications.
Updated by Victor Julien about 12 years ago
- Status changed from New to Assigned
- Assignee set to Ignacio Sanchez
- Target version changed from TBD to 1.4beta1
Overall, great work. I think this will be useful to many!
Code comments:
- check the result of SCMalloc
- use strlcpy instead of strncpy (no need to have the extra line to set the \0 anymore then)
- we have a loose line length limit of ~80
- please check small style things, like spaces between a = b;, instead of a=b;
Updated by Ignacio Sanchez about 12 years ago
- File log-httplog.c log-httplog.c added
- % Done changed from 0 to 20
Thank you for the comments.
Here goes a new version (patched log-httplog.c file).
If the style is OK I can send the git patch.
Updated by Victor Julien about 12 years ago
Looking a lot better, thanks!
Question: how does the %{X-Forwarded-For} deal with case? Will it match "x-forwarded-for" or "X-FORWARDED-FOR"?
Updated by Ignacio Sanchez about 12 years ago
Question: how does the %{X-Forwarded-For} deal with case? Will it match "x-forwarded-for" or "X-FORWARDED-FOR"?
Yes. It is not case sensitive. This is handled by the libhtp library.
According to RFC 2616 the HTTP headers are not case sensitive.
"Each header field consists of a name followed by a colon (":") and the field value. Field names are case-insensitive."
Updated by Victor Julien about 12 years ago
Makes sense, thanks! Can you attach a new git patch?
@Anoop can you do a full review of the new patch once it's attached?
Updated by Ignacio Sanchez about 12 years ago
Updated by Ignacio Sanchez about 12 years ago
- File deleted (
0001-Custom-logging-feature-for-log-httplog.patch)
Updated by Ignacio Sanchez about 12 years ago
This new patch (0003-XFF-support-for-alert-debuglog.patch) adds support for XFF in the alert-debuglog.
Via suricata yaml file it is possible to configure its behavior, either the XFF field is added to the alert or the source IP of the attack in the alert is overwritten by the last IP extracted from the XFF field of the http request which triggered the alert.
The XFF support is enabled using "xff: yes" under the alert-debug in the yaml file.
The XFF header name can be changed using "xffheadername". By default alert-debug will add the XFF content to the alert, but it is possible to overwrite the original source IP using "xffoverwrite: yes".
Example:
- a full alerts log containing much information for signature writers
- or for investigating suspected false positives.
- alert-debug:
enabled: yes
filename: alert-debug.log
append: yes
xff: yes
xffheadername: X-Forwarded-For
xffoverwrite: no
Updated by Victor Julien about 12 years ago
Why wouldn't you just print the xff header in the alert-debug log if it is available? These extra options seem unnecessary in that case.
Updated by Ignacio Sanchez about 12 years ago
Why wouldn't you just print the xff header in the alert-debug log if it is available? These extra options seem unnecessary in that case.
I implemented it that way as a test for the unified2 XFF support. I wanted the debuglog option mimic the ones of unified2. I considered that since barnyard2 does not currently support the XFF unified2 records, it could be useful to have suricata doing the replacement of the SRC IP (after all the logged packet is a fake one created by suricata). In this case the user could test it first in debuglog, or somebody might even use the output of debuglog directly into the SIEM without barnyard2 and unified2, if performance is not an issue.
In any case, I think we should split this feature into 2, custom http logging, and xff support. In my opinion the custom http logging (patches 0001 and 0002) are really to be included in the next suricata version and do not really depend on the XFF support.
Updated by Victor Julien about 12 years ago
Can you rebase and resubmit the log-http patches? (or do a pull request against https://github.com/inliniac/suricata)
Updated by Ignacio Sanchez about 12 years ago
- File 0001-Custom-logging-feature-for-log-httplog_REBASED.patch 0001-Custom-logging-feature-for-log-httplog_REBASED.patch added
Here it is (0001-Custom-logging-feature-for-log-httplog_REBASED.patch)
Victor Julien wrote:
Can you rebase and resubmit the log-http patches? (or do a pull request against https://github.com/inliniac/suricata)
Updated by Victor Julien about 12 years ago
- Target version changed from 1.4beta1 to 1.4
Updated by Ignacio Sanchez almost 12 years ago
- File u2-test-patchv2.diff u2-test-patchv2.diff added
- % Done changed from 50 to 90
I attach the XFF patch (u2-test-patchv2.diff) I have developed for the unified2 output.
It seems to be working OK in my tests. In the end it supports both the usage of the unified2 extra headers (like snort) and the IP replacement in the alert (so that the internal IP of the reverse proxy is replaced by the actual client IP taken from the XFF header).
The suricata.yaml configuration supports the usage of xff, xffheadername and xffoverwrite under the unified2-alert section.
Example:
- unified2-alert:
enabled: yes
filename: unified2.alert
xff: yes
xffheadername: X-Forwarded-For
xffoverwrite: yes
xffoverwrite specified the mode of operation. If it is set to yes, the alert IP address will be overwritten by the IP taken from the XFF header of the request which triggered the alert.
Updated by Victor Julien almost 12 years ago
Would you mind submitting this as a pull request at github? It has a nice review tool. See Git work flow.
Updated by Victor Julien almost 12 years ago
- Target version changed from 1.4 to TBD
1.4 release won't wait for this, but I will merge as soon as the code is ready. Some comments in the PR still have to be addressed.
Updated by Victor Julien almost 11 years ago
- Status changed from Assigned to Closed
- Target version changed from TBD to 2.0beta2
- % Done changed from 90 to 100
This has been merged into the git master, thanks to all involved!