Suricata

Adding the xff header to the http.log will be trivial I think. Chris, you interested in taking that on?

Wrt Unified2, we probably first need to get our u2 updated. We use a slightly older file format currently.

Ok. I attach a quick patch I have developed to add the X-Forwarded-For header, if present, to the extended http log.

I am considering adding a feature to allow the user specify a custom http log format via the yaml configuration file. This way, the user could customize the order, separator and the HTTP fields to be included in the http log.

What do you think?

Ok, I have created the following patch for the log-httplog which adds support for custom http logging using a format syntax inspired by Apache mod_log_config.

In order to activate the custom logging feature, the parameters custom and customformat shall be specified in the suricata.yaml configuration file.

Example (next to "extended" under http-log:

custom: yes # enable the custom logging format (defined by customformat)
customformat: "%{%D-%H:%M:%S}t.%z %{X-Forwarded-For}i %H %m %h %u %s %B %a:%p -> %A:%P"

In addition to %h, %H, %m, %u, %i, %C, %s, %o and %B - almost - as described by mod_log_config (http://httpd.apache.org/docs/2.0/mod/mod_log_config.html), I have added %z, %a, %p, %A and %P for the precision time, IPs and ports.

I have tested it in suricata 1.3.1b2 and in the latest suricata git version at the time of writing and it seems to be working fine.

As illustrated by the example, the XFF client IP can be logged with "%{X-Forwarded-For}i" and using the right customformat string the HTTP transaction log files would be directly readable by awstats or piwik so now we can have real time statistics of the monitored web applications.

Overall, great work. I think this will be useful to many!

Code comments:

- check the result of SCMalloc
- use strlcpy instead of strncpy (no need to have the extra line to set the \0 anymore then)
- we have a loose line length limit of ~80
- please check small style things, like spaces between a = b;, instead of a=b;

Thank you for the comments.

Here goes a new version (patched log-httplog.c file).

If the style is OK I can send the git patch.

This new patch (0003-XFF-support-for-alert-debuglog.patch) adds support for XFF in the alert-debuglog.

Via suricata yaml file it is possible to configure its behavior, either the XFF field is added to the alert or the source IP of the attack in the alert is overwritten by the last IP extracted from the XFF field of the http request which triggered the alert.

The XFF support is enabled using "xff: yes" under the alert-debug in the yaml file.

The XFF header name can be changed using "xffheadername". By default alert-debug will add the XFF content to the alert, but it is possible to overwrite the original source IP using "xffoverwrite: yes".

Example:

1. a full alerts log containing much information for signature writers
2. or for investigating suspected false positives.
   - alert-debug:
   enabled: yes
   filename: alert-debug.log
   append: yes
   xff: yes
   xffheadername: X-Forwarded-For
   xffoverwrite: no

Here it is (0001-Custom-logging-feature-for-log-httplog_REBASED.patch)

Victor Julien wrote:

Can you rebase and resubmit the log-http patches? (or do a pull request against https://github.com/inliniac/suricata)

I attach the XFF patch (u2-test-patchv2.diff) I have developed for the unified2 output.

It seems to be working OK in my tests. In the end it supports both the usage of the unified2 extra headers (like snort) and the IP replacement in the alert (so that the internal IP of the reverse proxy is replaced by the actual client IP taken from the XFF header).

The suricata.yaml configuration supports the usage of xff, xffheadername and xffoverwrite under the unified2-alert section.

Example:

- unified2-alert:
      enabled: yes
      filename: unified2.alert
      xff: yes
      xffheadername: X-Forwarded-For
      xffoverwrite: yes

xffoverwrite specified the mode of operation. If it is set to yes, the alert IP address will be overwritten by the IP taken from the XFF header of the request which triggered the alert.

1.4 release won't wait for this, but I will merge as soon as the code is ready. Some comments in the PR still have to be addressed.

This has been merged into the git master, thanks to all involved!

https://github.com/inliniac/suricata/pull/544