Suricata

Your unmodified rule fires for me here:

This is Suricata version 4.0.0 RELEASE
27/12/2017 -- 09:49:02 - <Notice> - This is Suricata version 4.0.0 RELEASE
27/12/2017 -- 09:49:02 - <Notice> - Ring buffer initialized with 5 files.
27/12/2017 -- 09:49:02 - <Notice> - all 5 packet processing threads, 4 management threads initialized, engine started.
27/12/2017 -- 09:49:02 - <Notice> - Signal Received.  Stopping engine.
27/12/2017 -- 09:49:02 - <Notice> - Pcap-file module read 89 packets, 159659 bytes
12/19/2017-13:01:34.001543  [**] [1:123456:1] MZ and CreateFont [**] [Classification: (null)] [Priority: 3] {TCP} 10.1.1.9:59724 -> 10.2.2.5:25

Here is the SMTP section of my yaml:

    smtp:
      enabled: yes
      # Configure SMTP-MIME Decoder
      mime:
        # Decode MIME messages from SMTP transactions
        # (may be resource intensive)
        # This field supercedes all others because it turns the entire
        # process on or off
        decode-mime: yes

        # Decode MIME entity bodies (ie. base64, quoted-printable, etc.)
        decode-base64: yes
        decode-quoted-printable: yes

        # Maximum bytes per header data value stored in the data structure
        # (default is 2000)
        header-value-depth: 2000

        # Extract URLs and save in state data structure
        extract-urls: yes
        # Set to yes to compute the md5 of the mail body. You will then
        # be able to journalize it.
        body-md5: yes
      # Configure inspected-tracker for file_data keyword
      inspected-tracker:
        content-limit: 100000
        content-inspect-min-size: 32768
        content-inspect-window: 4096

CreateFontIndirectW is at offset 11432, so I imagine your limits are lower than mine. See if you can tweak those to get it to fire.

Bryant Smith wrote:

I had noticed that when needing to inspect SMTP traffic that suricata can use file_data to inspect the base64 encoded attachments. This doesn't seem to work correctly all of the time. It seems that anything that is small seems to work but if the attachment is large I can't inspect deep into the payload or sometimes not even at the beginning of the payload. I've attached a sample pcap and simple rule that looks for two things. MZ at the beginning of the file_data payload and CreateFont which shows up further in. I've tried adjusting the various settings for libhtp but ended up with the same results.

Here is what I have for those values. Most of what is in my yaml file is default. Would there be something else affecting it? I've tried adjusting the libhtp values from kb to mb but that didn't seem to change anything.

    smtp:
      enabled: yes
      # Configure SMTP-MIME Decoder
      mime:
        # Decode MIME messages from SMTP transactions
        # (may be resource intensive)
        # This field supercedes all others because it turns the entire
        # process on or off
        decode-mime: yes

        # Decode MIME entity bodies (ie. base64, quoted-printable, etc.)
        decode-base64: yes
        decode-quoted-printable: yes

        # Maximum bytes per header data value stored in the data structure
        # (default is 2000)
        header-value-depth: 2000

        # Extract URLs and save in state data structure
        extract-urls: yes
        # Set to yes to compute the md5 of the mail body. You will then
        # be able to journalize it.
        body-md5: no
      # Configure inspected-tracker for file_data keyword
      inspected-tracker:
        content-limit: 100000
        content-inspect-min-size: 32768
        content-inspect-window: 4096

Bryant Smith wrote:

Here is what I have for those values. Most of what is in my yaml file is default. Would there be something else affecting it? I've tried adjusting the libhtp values from kb to mb but that didn't seem to change anything.

[...]

In the yaml you sent me it doesn't look like you are loading rules. Uncomment this:

# - test.rules

On line 52.

If that alone doesn't work, add this to your rules file (it appears you are using /opt/suricata/etc/suricata/rules/test.rules):

alert tcp any any -> any any (msg:"FILE store all"; filestore; flowbits:noalert; sid:303; rev:1;)

If I turn off that filestore rule it doesn't fire locally. I am not sure if that's expected behavior.

I am loading rule but I don't use the yaml file to load them. I use the -S option in a small script to run pcaps and different rule files. I have multiple rule files that I test with and this seemed to be the easiest way to run it.

/opt/suricata/bin/suricata -c /opt/suricata/etc/suricata/suricata.yaml -k none -r $PCAP -S $FILE --runmode=single

So I added your filestore signature to the rule file I was using and I got an alert from it. I'm not sure why this works though, any ideas?

12/19/2017-13:01:34.001543  [**] [1:123456:1] MZ and CreateFont [**] [Classification: (null)] [Priority: 3] {TCP} 10.1.1.9:59724 -> 10.2.2.5:25

  --------------------------------------------------------------------------
  Date: 12/28/2017 -- 07:24:33. Sorted by: number of matches.
  --------------------------------------------------------------------------
   Num      Rule         Gid      Rev      Ticks        %      Checks   Matches  Max Ticks   Avg Ticks   Avg Match   Avg No Match
  -------- ------------ -------- -------- ------------ ------ -------- -------- ----------- ----------- ----------- -------------- 
  1        123456       1        1        45674        10.09  1        1        45674       45674.00    45674.00    0.00       
  2        303          1        1        407015       89.91  166      0        51319       2451.90     0.00        2451.90

That I do not know. I'll see what I can find out.

Bryant Smith wrote:

So I added your filestore signature to the rule file I was using and I got an alert from it. I'm not sure why this works though, any ideas?

[...]

I went in and made some changes to the yaml file with the following. If I turn on the file-store and enable force-filestore then the signature will work. This is probably the same as the siganture

  - file-store:
      enabled: yes       # set to yes to enable
      log-dir: files    # directory to store the files
      force-magic: yes   # force logging magic on all stored files
      # force logging of checksums, available hash functions are md5,
      # sha1 and sha256
      #force-hash: [md5]
      force-filestore: yes # force storing of all files
      # override global stream-depth for sessions in which we want to
      # perform file extraction. Set to 0 for unlimited.
      #stream-depth: 0
      #waldo: file.waldo # waldo file to store the file_id across runs

@Gabriel what does your suricata.yaml look like?

I can reproduce it, even increasing the limits within libhtp settings didn't change. It triggers at least for the zip if I remove the part beginning with file_data. We might need to dig deeper into that why it's not matching.

Can someone create a suricata-verify test for it?

https://github.com/OISF/suricata-verify/pull/113

Opened new bug #3190 to keep things simple.