Project

General

Profile

Actions

Feature #478

closed

XFF (X-Forwarded-For)

Added by Ignacio Sanchez almost 12 years ago. Updated over 10 years ago.

Status:
Closed
Priority:
Normal
Target version:
Effort:
Difficulty:
Label:

Description

XFF support so that Suricata could be placed behind a reverse proxy which terminates the SSL and SNATs the HTTP connections to the web servers.

Currently, in this scenario Suricata sees all inbound HTTP connections as coming from the internal IP address of the reverse proxy. Therefore if an attack is detected, the reverse proxy IP address will be recorded as the attacker's IP. The http-log will also just display the reverse proxy IP.

Reverse proxies, when deployed in non transparent mode, typically insert the original client IP address into an HTTP header such as X-Forwarded-For.

Snort, since version 2.9.0 supports the extraction of this header (--enable_xff) and inserts it as a extrahdr record in the unified2 files.

Suggested features:

  • Insert into an extrahdr unified2 record the actual client IP address if it is present in the HTTP header of the HTTP request which triggered the alert. XFF feature and XFF header name could be configured via suricata's yaml configuration file.
  • Add client XFF IP address to the extended http-log (if XFF is configured via suricata's yaml configuration file).

Files

XFF_basic_extended-http-log.patch (885 Bytes) XFF_basic_extended-http-log.patch X-Forwarded-For (XFF) header Ignacio Sanchez, 06/13/2012 10:29 AM
log-httplog.c (26.9 KB) log-httplog.c Custom logging feature (patched log-httplog.c) Ignacio Sanchez, 07/05/2012 11:36 AM
0001-Custom-logging-feature-for-log-httplog.patch (18.3 KB) 0001-Custom-logging-feature-for-log-httplog.patch Custom logging feature patch #1 Ignacio Sanchez, 07/06/2012 05:50 AM
0002-SCMalloc-result-check-strncpy-replaced-by-strlcpy-im.patch (17 KB) 0002-SCMalloc-result-check-strncpy-replaced-by-strlcpy-im.patch Custom logging feature patch #2 (fixing Victor's comments) Ignacio Sanchez, 07/06/2012 05:50 AM
0003-XFF-support-for-alert-debuglog.patch (8.87 KB) 0003-XFF-support-for-alert-debuglog.patch XFF support for alert-debug Ignacio Sanchez, 08/07/2012 06:02 AM
0001-Custom-logging-feature-for-log-httplog_REBASED.patch (19.4 KB) 0001-Custom-logging-feature-for-log-httplog_REBASED.patch Custom logging feature for log-httplog (rebased) Ignacio Sanchez, 08/22/2012 10:53 AM
u2-test-patchv2.diff (20.7 KB) u2-test-patchv2.diff Ignacio Sanchez, 09/28/2012 08:32 AM

Related issues 2 (0 open2 closed)

Related to Suricata - Feature #38: Configuration for custom log format for HTTP logClosedEric Leblond01/02/2010Actions
Related to Suricata - Feature #530: Custom http loggingClosedIgnacio Sanchez08/23/2012Actions
Actions #1

Updated by Victor Julien almost 12 years ago

  • Target version set to TBD

Adding the xff header to the http.log will be trivial I think. Chris, you interested in taking that on?

Wrt Unified2, we probably first need to get our u2 updated. We use a slightly older file format currently.

Actions #2

Updated by Ignacio Sanchez almost 12 years ago

Ok. I attach a quick patch I have developed to add the X-Forwarded-For header, if present, to the extended http log.

I am considering adding a feature to allow the user specify a custom http log format via the yaml configuration file. This way, the user could customize the order, separator and the HTTP fields to be included in the http log.

What do you think?

Actions #3

Updated by Victor Julien almost 12 years ago

Yeah this looks good. Might be interesting to add it to the file extraction/inspection related outputs as well, such as the files-json.log and the file store meta files.

Actions #4

Updated by Ignacio Sanchez almost 12 years ago

  • File 0001-Custom-logging-feature-for-log-httplog.patch added
  • File log-httplog.c added

Ok, I have created the following patch for the log-httplog which adds support for custom http logging using a format syntax inspired by Apache mod_log_config.

In order to activate the custom logging feature, the parameters custom and customformat shall be specified in the suricata.yaml configuration file.

Example (next to "extended" under http-log:

custom: yes # enable the custom logging format (defined by customformat)
customformat: "%{%D-%H:%M:%S}t.%z %{X-Forwarded-For}i %H %m %h %u %s %B %a:%p -> %A:%P"

In addition to %h, %H, %m, %u, %i, %C, %s, %o and %B - almost - as described by mod_log_config (http://httpd.apache.org/docs/2.0/mod/mod_log_config.html), I have added %z, %a, %p, %A and %P for the precision time, IPs and ports.

I have tested it in suricata 1.3.1b2 and in the latest suricata git version at the time of writing and it seems to be working fine.

As illustrated by the example, the XFF client IP can be logged with "%{X-Forwarded-For}i" and using the right customformat string the HTTP transaction log files would be directly readable by awstats or piwik so now we can have real time statistics of the monitored web applications.

Actions #5

Updated by Victor Julien almost 12 years ago

  • Status changed from New to Assigned
  • Assignee set to Ignacio Sanchez
  • Target version changed from TBD to 1.4beta1

Overall, great work. I think this will be useful to many!

Code comments:

- check the result of SCMalloc
- use strlcpy instead of strncpy (no need to have the extra line to set the \0 anymore then)
- we have a loose line length limit of ~80
- please check small style things, like spaces between a = b;, instead of a=b;

Actions #6

Updated by Ignacio Sanchez almost 12 years ago

Thank you for the comments.

Here goes a new version (patched log-httplog.c file).

If the style is OK I can send the git patch.

Actions #7

Updated by Ignacio Sanchez almost 12 years ago

  • File deleted (log-httplog.c)
Actions #8

Updated by Victor Julien almost 12 years ago

Looking a lot better, thanks!

Question: how does the %{X-Forwarded-For} deal with case? Will it match "x-forwarded-for" or "X-FORWARDED-FOR"?

Actions #9

Updated by Ignacio Sanchez almost 12 years ago

Question: how does the %{X-Forwarded-For} deal with case? Will it match "x-forwarded-for" or "X-FORWARDED-FOR"?

Yes. It is not case sensitive. This is handled by the libhtp library.

According to RFC 2616 the HTTP headers are not case sensitive.

"Each header field consists of a name followed by a colon (":") and the field value. Field names are case-insensitive."

Actions #10

Updated by Victor Julien almost 12 years ago

Makes sense, thanks! Can you attach a new git patch?

@Anoop can you do a full review of the new patch once it's attached?

Actions #11

Updated by Anoop Saldanha almost 12 years ago

Sure.

Actions #13

Updated by Ignacio Sanchez almost 12 years ago

  • File deleted (0001-Custom-logging-feature-for-log-httplog.patch)
Actions #14

Updated by Ignacio Sanchez almost 12 years ago

This new patch (0003-XFF-support-for-alert-debuglog.patch) adds support for XFF in the alert-debuglog.

Via suricata yaml file it is possible to configure its behavior, either the XFF field is added to the alert or the source IP of the attack in the alert is overwritten by the last IP extracted from the XFF field of the http request which triggered the alert.

The XFF support is enabled using "xff: yes" under the alert-debug in the yaml file.

The XFF header name can be changed using "xffheadername". By default alert-debug will add the XFF content to the alert, but it is possible to overwrite the original source IP using "xffoverwrite: yes".

Example:

  1. a full alerts log containing much information for signature writers
  2. or for investigating suspected false positives.
    - alert-debug:
    enabled: yes
    filename: alert-debug.log
    append: yes
    xff: yes
    xffheadername: X-Forwarded-For
    xffoverwrite: no
Actions #15

Updated by Ignacio Sanchez almost 12 years ago

  • % Done changed from 20 to 50
Actions #16

Updated by Victor Julien almost 12 years ago

Why wouldn't you just print the xff header in the alert-debug log if it is available? These extra options seem unnecessary in that case.

Actions #17

Updated by Ignacio Sanchez almost 12 years ago

Why wouldn't you just print the xff header in the alert-debug log if it is available? These extra options seem unnecessary in that case.

I implemented it that way as a test for the unified2 XFF support. I wanted the debuglog option mimic the ones of unified2. I considered that since barnyard2 does not currently support the XFF unified2 records, it could be useful to have suricata doing the replacement of the SRC IP (after all the logged packet is a fake one created by suricata). In this case the user could test it first in debuglog, or somebody might even use the output of debuglog directly into the SIEM without barnyard2 and unified2, if performance is not an issue.

In any case, I think we should split this feature into 2, custom http logging, and xff support. In my opinion the custom http logging (patches 0001 and 0002) are really to be included in the next suricata version and do not really depend on the XFF support.

Actions #18

Updated by Victor Julien over 11 years ago

Can you rebase and resubmit the log-http patches? (or do a pull request against https://github.com/inliniac/suricata)

Actions #19

Updated by Ignacio Sanchez over 11 years ago

Here it is (0001-Custom-logging-feature-for-log-httplog_REBASED.patch)

Victor Julien wrote:

Can you rebase and resubmit the log-http patches? (or do a pull request against https://github.com/inliniac/suricata)

Actions #20

Updated by Victor Julien over 11 years ago

  • Target version changed from 1.4beta1 to 1.4
Actions #21

Updated by Ignacio Sanchez over 11 years ago

I attach the XFF patch (u2-test-patchv2.diff) I have developed for the unified2 output.

It seems to be working OK in my tests. In the end it supports both the usage of the unified2 extra headers (like snort) and the IP replacement in the alert (so that the internal IP of the reverse proxy is replaced by the actual client IP taken from the XFF header).

The suricata.yaml configuration supports the usage of xff, xffheadername and xffoverwrite under the unified2-alert section.

Example:

- unified2-alert:
enabled: yes
filename: unified2.alert
xff: yes
xffheadername: X-Forwarded-For
xffoverwrite: yes

xffoverwrite specified the mode of operation. If it is set to yes, the alert IP address will be overwritten by the IP taken from the XFF header of the request which triggered the alert.

Actions #22

Updated by Victor Julien over 11 years ago

Would you mind submitting this as a pull request at github? It has a nice review tool. See Git work flow.

Actions #23

Updated by Ignacio Sanchez over 11 years ago

Ok. Done.

Actions #24

Updated by Victor Julien over 11 years ago

  • Target version changed from 1.4 to TBD

1.4 release won't wait for this, but I will merge as soon as the code is ready. Some comments in the PR still have to be addressed.

Actions #25

Updated by Victor Julien over 10 years ago

  • Status changed from Assigned to Closed
  • Target version changed from TBD to 2.0beta2
  • % Done changed from 90 to 100

This has been merged into the git master, thanks to all involved!

https://github.com/inliniac/suricata/pull/544

Actions

Also available in: Atom PDF