Project

General

Profile

Do not edit. This page has been migrated. See Converting_Wiki_Documentation_to_Sphinx.

Storing MD5s checksums

In this particular example we are using: Ubuntu 14.04 LTS

Also - we are using the latest git master (git installation)

Make sure you have libnss and libnspr installed

root@LTS-64-1:~/Work/tmp/oisf# dpkg -l |grep libnss
ii  libnss-mdns:amd64                                     0.10-6                                              amd64        NSS module for Multicast DNS name resolution
ii  libnss3:amd64                                         2:3.17.4-0ubuntu0.14.04.1                           amd64        Network Security Service libraries
ii  libnss3-1d:amd64                                      2:3.17.4-0ubuntu0.14.04.1                           amd64        Network Security Service libraries - transitional package
ii  libnss3-dev:amd64                                     2:3.17.4-0ubuntu0.14.04.1                           amd64        Development files for the Network Security Service libraries
ii  libnss3-nssdb                                         2:3.17.4-0ubuntu0.14.04.1                           all          Network Security Security libraries - shared databases
ii  libnss3-tools                                         2:3.17.4-0ubuntu0.14.04.1                           amd64        Network Security Service tools

root@LTS-64-1:~/Work/tmp/oisf# dpkg -l |grep libnspr
ii  libnspr4:amd64                                        2:4.10.7-0ubuntu0.14.04.1                           amd64        NetScape Portable Runtime Library
ii  libnspr4-dev                                          2:4.10.7-0ubuntu0.14.04.1                           amd64        Development files for the NetScape Portable Runtime library

If not install them:

apt-get install libnss3-dev libnspr4-dev

Note: Fedora users need to install the following:

nss-util
nss-util-devel
nss-devel
nspr-devel
nspr

Get the Suricata code

Execute:

git clone git://phalanx.openinfosecfoundation.org/oisf.git && cd oisf
git clone https://github.com/OISF/libhtp.git -b 0.5.x

Building Suricata

You have to compile/install suri like this in order to enable MD5s:

./autogen.sh
./configure --with-libnss-libraries=/usr/lib --with-libnss-includes=/usr/include/nss/ --with-libnspr-libraries=/usr/lib --with-libnspr-includes=/usr/include/nspr
make clean
make
sudo make install

Output of configure:

Suricata Configuration:
   AF_PACKET support:                       yes
   PF_RING support:                         no
   NFQueue support:                         no
   IPFW support:                            no
   DAG enabled:                             no
   Napatech enabled:                        no

   libnss support:                          yes
   libnspr support:                         yes
   Prelude support:                         no
   PCRE jit:                                no

This is what is important to have:

libnss support:                          yes
libnspr support:                         yes

Confirm everything is built correctly:

# suricata --build-info
[10010] 1/5/2012 -- 11:16:23 - (suricata.c:502) <Info> (SCPrintBuildInfo) -- This is Suricata version 1.3dev (rev e6dea5c)
[10010] 1/5/2012 -- 11:16:23 - (suricata.c:575) <Info> (SCPrintBuildInfo) -- Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK HAVE_HTP_TX_GET_RESPONSE_HEADERS_RAW HAVE_NSS
[10010] 1/5/2012 -- 11:16:23 - (suricata.c:589) <Info> (SCPrintBuildInfo) -- 32-bits, Little-endian architecture
[10010] 1/5/2012 -- 11:16:23 - (suricata.c:591) <Info> (SCPrintBuildInfo) -- GCC version 4.4.5, C version 199901
[10010] 1/5/2012 -- 11:16:23 - (suricata.c:597) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1
[10010] 1/5/2012 -- 11:16:23 - (suricata.c:600) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2
[10010] 1/5/2012 -- 11:16:23 - (suricata.c:603) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4
[10010] 1/5/2012 -- 11:16:23 - (suricata.c:606) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8
[10010] 1/5/2012 -- 11:16:23 - (suricata.c:613) <Info> (SCPrintBuildInfo) -- compiled with -fstack-protector
[10010] 1/5/2012 -- 11:16:23 - (suricata.c:619) <Info> (SCPrintBuildInfo) -- compiled with _FORTIFY_SOURCE=2

Make sure we have HAVE_NSS in the Features line.

Configuration

In the suricata yaml:

  - file-store:
       enabled: yes       # set to yes to enable
       log-dir: files     # directory to store the files
       force-magic: yes   # force logging magic on all stored files
       force-md5: yes     # force logging of md5 checksums
       #waldo: file.waldo # waldo file to store the file_id across runs

Optionally, for JSON output:

 - file-log:
   enabled: yes
   filename: files-json.log
   append: no

Other settings affecting File Extraction

stream:
  memcap: 64mb
  checksum-validation: yes      # reject wrong csums
  inline: no                    # no inline mode
  reassembly:
    memcap: 32mb
    depth: 0                     # reassemble all of a stream
    toserver-chunk-size: 2560
    toclient-chunk-size: 2560

Make sure we have depth: 0 so all files can be tracked fully.

libhtp:
  default-config:
    personality: IDS
    # Can be specified in kb, mb, gb.  Just a number indicates
    # it's in bytes.
    request-body-limit: 0
    response-body-limit: 0

Make sure we have request-body-limit: 0 and response-body-limit: 0

Testing

For the purpose of testing we use this rule only in a file.rules (a test/example file):

alert http any any -> any any (msg:"FILE store all"; filestore; sid:1; rev:1;)

This rule above will save all the file data for files that are opened/downloaded through HTTP

Start Suricta (-S option loads ONLY the specified rule file, with disregard if any other rules that are enabled in suricata.yaml):

suricata -c /etc/suricata/suricata.yaml -S file.rules -i eth0

I tried that link (Cisco Prod Brochure PDF, just googled "Cisco PDF"): http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CDAQFjAA&url=http%3A%2F%2Fwww.cisco.com%2Fen%2FUS%2Fprod%2Fcollateral%2Frouters%2Fps5855%2Fprod_brochure0900aecd8019dc1f.pdf&ei=OqyfT9eoJubi4QTyiamhAw&usg=AFQjCNGdjDBpBDfQv2r3VogSH41V6T5x9Q

and in file directory (/var/log/suricata/files) I got the meta data:

TIME:              05/01/2012-11:09:52.425751
SRC IP:            2.23.144.170
DST IP:            192.168.1.91
PROTO:             6
SRC PORT:          80
DST PORT:          51598
HTTP URI:          /en/US/prod/collateral/routers/ps5855/prod_brochure0900aecd8019dc1f.pdf
HTTP HOST:         www.cisco.com
HTTP REFERER:      http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CDAQFjAA&url=http%3A%2F%2Fwww.cisco.com%2Fen%2FUS%2Fprod%2Fcollateral%2Frouters%2Fps5855%2Fprod_brochure0900aecd8019dc1f.pdf&ei=OqyfT9eoJubi4QTyiamhAw&usg=AFQjCNGdjDBpBDfQv2r3VogSH41V6T5x9Q
FILENAME:          /en/US/prod/collateral/routers/ps5855/prod_brochure0900aecd8019dc1f.pdf
MAGIC:             PDF document, version 1.6
STATE:             CLOSED
MD5:               59eba188e52467adc11bf2442ee5bf57
SIZE:              9485123

and in files-json.log (or eve.json) :

{ "id": 1, "timestamp": "05\/01\/2012-11:10:27.693583", "ipver": 4, "srcip": "2.23.144.170", "dstip": "192.168.1.91", "protocol": 6, "sp": 80, "dp": 51598, "http_uri": "\/en\/US\/prod\/collateral\/routers\/ps5855\/prod_brochure0900aecd8019dc1f.pdf", "http_host": "www.cisco.com", "http_referer": "http:\/\/www.google.com\/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CDAQFjAA&url=http%3A%2F%2Fwww.cisco.com%2Fen%2FUS%2Fprod%2Fcollateral%2Frouters%2Fps5855%2Fprod_brochure0900aecd8019dc1f.pdf&ei=OqyfT9eoJubi4QTyiamhAw&usg=AFQjCNGdjDBpBDfQv2r3VogSH41V6T5x9Q", "filename": "\/en\/US\/prod\/collateral\/routers\/ps5855\/prod_brochure0900aecd8019dc1f.pdf", "magic": "PDF document, version 1.6", "state": "CLOSED", "md5": "59eba188e52467adc11bf2442ee5bf57", "stored": true, "size": 9485123 }
{ "id": 12, "timestamp": "05\/01\/2012-11:12:57.421420", "ipver": 4, "srcip": "2.23.144.170", "dstip": "192.168.1.91", "protocol": 6, "sp": 80, "dp": 51598, "http_uri": "\/en\/US\/prod\/collateral\/routers\/ps5855\/prod_brochure0900aecd8019dc1f.pdf", "http_host": "www.cisco.com", "http_referer": "http:\/\/www.google.com\/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CDAQFjAA&url=http%3A%2F%2Fwww.cisco.com%2Fen%2FUS%2Fprod%2Fcollateral%2Frouters%2Fps5855%2Fprod_brochure0900aecd8019dc1f.pdf&ei=OqyfT9eoJubi4QTyiamhAw&usg=AFQjCNGdjDBpBDfQv2r3VogSH41V6T5x9Q", "filename": "\/en\/US\/prod\/collateral\/routers\/ps5855\/prod_brochure0900aecd8019dc1f.pdf", "magic": "PDF document, version 1.6", "state": "CLOSED", "md5": "59eba188e52467adc11bf2442ee5bf57", "stored": true, "size": 9485123 }

Log all MD5s without any rules

If you would like to log MD5s for everything and anything that passes through the traffic that you are inspecting with Suricata, but not log the files themselves, all you have to do is disable file-store and enable only the JSON output with forced MD5s - in suricata.yaml like so:

- file-store:
    enabled: no       # set to yes to enable
    log-dir: files    # directory to store the files
    force-magic: yes   # force logging magic on all stored files
    force-md5: yes     # force logging of md5 checksums
    #waldo: file.waldo # waldo file to store the file_id across runs

- file-log:
    enabled: yes
    filename: files-json.log
    append: no
    #filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'
    force-magic: yes   # force logging magic on all logged files
    force-md5: yes     # force logging of md5 checksums

This is in short what is needed to have MD5s logged.