Project

General

Profile

Actions

Bug #2395

closed

File_data inspection depth while inspecting base64 decoded data

Added by Bryant Smith about 5 years ago. Updated over 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

I had noticed that when needing to inspect SMTP traffic that suricata can use file_data to inspect the base64 encoded attachments. This doesn't seem to work correctly all of the time. It seems that anything that is small seems to work but if the attachment is large I can't inspect deep into the payload or sometimes not even at the beginning of the payload. I've attached a sample pcap and simple rule that looks for two things. MZ at the beginning of the file_data payload and CreateFont which shows up further in. I've tried adjusting the various settings for libhtp but ended up with the same results.

/opt/suricata/bin/suricata -V

This is Suricata version 4.0.0 RELEASE

--------------------------------------------------------------------------
Date: 12/26/2017 -- 07:18:17. Sorted by: number of matches.
--------------------------------------------------------------------------
Num Rule Gid Rev Ticks % Checks Matches Max Ticks Avg Ticks Avg Match Avg No Match
-------- ------------ -------- -------- ------------ ------ -------- -------- ----------- ----------- ----------- --------------
1 123456 1 1 26488 100.00 1 0 26488 26488.00 0.00 26488.00

Files

sample.rules (212 Bytes) sample.rules Bryant Smith, 12/26/2017 08:11 AM
export.pcap (157 KB) export.pcap Bryant Smith, 12/26/2017 08:11 AM
test-mail-attach.pcap (5.69 MB) test-mail-attach.pcap Gabriel Somlo, 09/24/2018 05:03 PM
etc_suricata_suricata.yaml (72.8 KB) etc_suricata_suricata.yaml Gabriel Somlo, 08/19/2019 11:16 PM
etc_suricata_classification.config (4.16 KB) etc_suricata_classification.config Gabriel Somlo, 08/19/2019 11:16 PM
var_lib_suricata_rules_suricata.rules (315 Bytes) var_lib_suricata_rules_suricata.rules Gabriel Somlo, 08/19/2019 11:16 PM
suricata.rules (436 Bytes) suricata.rules Gabriel Somlo, 09/22/2019 01:00 PM

Related issues 2 (0 open2 closed)

Related to Bug #3190: file_data inspection inhibited by additional (non-file_data) content match ruleClosedVictor JulienActions
Copied to Bug #3175: File_data inspection depth while inspecting base64 decoded data (4.1.x)ClosedVictor JulienActions
Actions #1

Updated by Francis Trudeau about 5 years ago

Your unmodified rule fires for me here:


This is Suricata version 4.0.0 RELEASE
27/12/2017 -- 09:49:02 - <Notice> - This is Suricata version 4.0.0 RELEASE
27/12/2017 -- 09:49:02 - <Notice> - Ring buffer initialized with 5 files.
27/12/2017 -- 09:49:02 - <Notice> - all 5 packet processing threads, 4 management threads initialized, engine started.
27/12/2017 -- 09:49:02 - <Notice> - Signal Received.  Stopping engine.
27/12/2017 -- 09:49:02 - <Notice> - Pcap-file module read 89 packets, 159659 bytes
12/19/2017-13:01:34.001543  [**] [1:123456:1] MZ and CreateFont [**] [Classification: (null)] [Priority: 3] {TCP} 10.1.1.9:59724 -> 10.2.2.5:25

Here is the SMTP section of my yaml:

    smtp:
      enabled: yes
      # Configure SMTP-MIME Decoder
      mime:
        # Decode MIME messages from SMTP transactions
        # (may be resource intensive)
        # This field supercedes all others because it turns the entire
        # process on or off
        decode-mime: yes

        # Decode MIME entity bodies (ie. base64, quoted-printable, etc.)
        decode-base64: yes
        decode-quoted-printable: yes

        # Maximum bytes per header data value stored in the data structure
        # (default is 2000)
        header-value-depth: 2000

        # Extract URLs and save in state data structure
        extract-urls: yes
        # Set to yes to compute the md5 of the mail body. You will then
        # be able to journalize it.
        body-md5: yes
      # Configure inspected-tracker for file_data keyword
      inspected-tracker:
        content-limit: 100000
        content-inspect-min-size: 32768
        content-inspect-window: 4096

CreateFontIndirectW is at offset 11432, so I imagine your limits are lower than mine. See if you can tweak those to get it to fire.

Bryant Smith wrote:

I had noticed that when needing to inspect SMTP traffic that suricata can use file_data to inspect the base64 encoded attachments. This doesn't seem to work correctly all of the time. It seems that anything that is small seems to work but if the attachment is large I can't inspect deep into the payload or sometimes not even at the beginning of the payload. I've attached a sample pcap and simple rule that looks for two things. MZ at the beginning of the file_data payload and CreateFont which shows up further in. I've tried adjusting the various settings for libhtp but ended up with the same results.

Actions #2

Updated by Bryant Smith about 5 years ago

Here is what I have for those values. Most of what is in my yaml file is default. Would there be something else affecting it? I've tried adjusting the libhtp values from kb to mb but that didn't seem to change anything.

    smtp:
      enabled: yes
      # Configure SMTP-MIME Decoder
      mime:
        # Decode MIME messages from SMTP transactions
        # (may be resource intensive)
        # This field supercedes all others because it turns the entire
        # process on or off
        decode-mime: yes

        # Decode MIME entity bodies (ie. base64, quoted-printable, etc.)
        decode-base64: yes
        decode-quoted-printable: yes

        # Maximum bytes per header data value stored in the data structure
        # (default is 2000)
        header-value-depth: 2000

        # Extract URLs and save in state data structure
        extract-urls: yes
        # Set to yes to compute the md5 of the mail body. You will then
        # be able to journalize it.
        body-md5: no
      # Configure inspected-tracker for file_data keyword
      inspected-tracker:
        content-limit: 100000
        content-inspect-min-size: 32768
        content-inspect-window: 4096

Actions #3

Updated by Francis Trudeau about 5 years ago

Bryant Smith wrote:

Here is what I have for those values. Most of what is in my yaml file is default. Would there be something else affecting it? I've tried adjusting the libhtp values from kb to mb but that didn't seem to change anything.

[...]

In the yaml you sent me it doesn't look like you are loading rules. Uncomment this:

# - test.rules

On line 52.

If that alone doesn't work, add this to your rules file (it appears you are using /opt/suricata/etc/suricata/rules/test.rules):

alert tcp any any -> any any (msg:"FILE store all"; filestore; flowbits:noalert; sid:303; rev:1;)

If I turn off that filestore rule it doesn't fire locally. I am not sure if that's expected behavior.

Actions #4

Updated by Bryant Smith about 5 years ago

I am loading rule but I don't use the yaml file to load them. I use the -S option in a small script to run pcaps and different rule files. I have multiple rule files that I test with and this seemed to be the easiest way to run it.

/opt/suricata/bin/suricata -c /opt/suricata/etc/suricata/suricata.yaml -k none -r $PCAP -S $FILE --runmode=single

Actions #5

Updated by Bryant Smith about 5 years ago

So I added your filestore signature to the rule file I was using and I got an alert from it. I'm not sure why this works though, any ideas?

12/19/2017-13:01:34.001543  [**] [1:123456:1] MZ and CreateFont [**] [Classification: (null)] [Priority: 3] {TCP} 10.1.1.9:59724 -> 10.2.2.5:25

  --------------------------------------------------------------------------
  Date: 12/28/2017 -- 07:24:33. Sorted by: number of matches.
  --------------------------------------------------------------------------
   Num      Rule         Gid      Rev      Ticks        %      Checks   Matches  Max Ticks   Avg Ticks   Avg Match   Avg No Match
  -------- ------------ -------- -------- ------------ ------ -------- -------- ----------- ----------- ----------- -------------- 
  1        123456       1        1        45674        10.09  1        1        45674       45674.00    45674.00    0.00       
  2        303          1        1        407015       89.91  166      0        51319       2451.90     0.00        2451.90
Actions #6

Updated by Francis Trudeau about 5 years ago

That I do not know. I'll see what I can find out.

Bryant Smith wrote:

So I added your filestore signature to the rule file I was using and I got an alert from it. I'm not sure why this works though, any ideas?

[...]

Actions #7

Updated by Bryant Smith about 5 years ago

I went in and made some changes to the yaml file with the following. If I turn on the file-store and enable force-filestore then the signature will work. This is probably the same as the siganture


  - file-store:
      enabled: yes       # set to yes to enable
      log-dir: files    # directory to store the files
      force-magic: yes   # force logging magic on all stored files
      # force logging of checksums, available hash functions are md5,
      # sha1 and sha256
      #force-hash: [md5]
      force-filestore: yes # force storing of all files
      # override global stream-depth for sessions in which we want to
      # perform file extraction. Set to 0 for unlimited.
      #stream-depth: 0
      #waldo: file.waldo # waldo file to store the file_id across runs

Actions #8

Updated by Andreas Herz about 5 years ago

  • Assignee set to Anonymous
  • Target version set to TBD
Actions #9

Updated by Gabriel Somlo over 4 years ago

I'm experiencing the same issue (failure to trigger on smtp base64 "file_data" pattern), both using the Fedora 28 build of version 4.0.5, and using a recent 4.1.0 rc -- the github master as of last week, commit 0b5a2ab4).

Using the default suricata.yaml, with missing rule files (mentioned but not installed by default) commented out, and with a single "local.rules" file added to the list, the following rule:

alert tcp $EXTERNAL_NET any -> $HOME_NET 25 (msg:"VIRUS INBOUND bad file attachment"; flow:to_server,established; content:"content-disposition|3a| attachment|3b|"; nocase; content:".zip|22|"; nocase; within:128; file_data; content:".pdf.exe"; within:64; classtype:suspicious-attachment-detect; sid:3000101; rev:1;)

fails to trigger on the attached test-mail-attach.pcap file. When I try it using snort, the same signature does find the match.

Actions #10

Updated by Andreas Herz almost 4 years ago

  • Assignee set to Community Ticket
Actions #11

Updated by Andreas Herz over 3 years ago

@Gabriel what does your suricata.yaml look like?

Updated by Gabriel Somlo over 3 years ago

Andreas Herz wrote:

@Gabriel what does your suricata.yaml look like?

@Andreas: I am now using fedora 30 with suricata-4.1.4-1.fc30.x86_64. I am uploading suricata.yaml and classification.config, as well as my (one-liner) suricata.rules file (containing the rule I referred to in comment #9. The changes I made to the default suricata.yaml are highlighted below:

--- suricata.yaml.orig  2019-08-19 15:50:30.794671681 -0400
+++ suricata.yaml       2019-08-19 15:50:41.561663720 -0400
@@ -10,18 +10,18 @@
 ##

 vars:
   # more specific is better for alert accuracy and performance
   address-groups:
-    HOME_NET: "[192.168.0.0/16,10.0.0.0/8,172.16.0.0/12]" 
+    #HOME_NET: "[192.168.0.0/16,10.0.0.0/8,172.16.0.0/12]" 
     #HOME_NET: "[192.168.0.0/16]" 
     #HOME_NET: "[10.0.0.0/8]" 
     #HOME_NET: "[172.16.0.0/12]" 
-    #HOME_NET: "any" 
+    HOME_NET: "any" 

-    EXTERNAL_NET: "!$HOME_NET" 
-    #EXTERNAL_NET: "any" 
+    #EXTERNAL_NET: "!$HOME_NET" 
+    EXTERNAL_NET: "any" 

     HTTP_SERVERS: "$HOME_NET" 
     SMTP_SERVERS: "$HOME_NET" 
     SQL_SERVERS: "$HOME_NET" 
     DNS_SERVERS: "$HOME_NET" 

The changes I made to classification.config:
--- classification.config.orig  2019-08-19 15:58:35.992307453 -0400
+++ classification.config       2019-08-19 16:00:03.886240250 -0400
@@ -73,5 +73,8 @@
 config classification: domain-c2,Domain Observed Used for C2 Detected,1
 config classification: pup-activity,Possibly Unwanted Program Detected,2
 config classification: credential-theft,Successful Credential Theft Detected,1
 config classification: social-engineering,Possible Social Engineering Attempted,2
 config classification: coin-mining,Crypto Currency Mining Activity Detected,2
+
+# NetFor
+config classification: suspicious-attachment-detect,Suspicious Attachment Detected,1

With this setup, running
suricata -r test-mail-attach.pcap

should generate a hit in /var/log/suricata/fast.log, but doesn't.

Actions #13

Updated by Andreas Herz over 3 years ago

I can reproduce it, even increasing the limits within libhtp settings didn't change. It triggers at least for the zip if I remove the part beginning with file_data. We might need to dig deeper into that why it's not matching.

Actions #14

Updated by Victor Julien over 3 years ago

Can someone create a suricata-verify test for it?

Actions #16

Updated by Andreas Herz over 3 years ago

  • Assignee changed from Community Ticket to Andreas Herz
Actions #17

Updated by Victor Julien over 3 years ago

  • Status changed from New to Assigned
  • Assignee changed from Andreas Herz to Victor Julien
  • Target version changed from TBD to 5.0rc1
Actions #19

Updated by Victor Julien over 3 years ago

  • Copied to Bug #3175: File_data inspection depth while inspecting base64 decoded data (4.1.x) added
Actions #20

Updated by Gabriel Somlo over 3 years ago

@Victor, thanks for the fix, it does indeed change the behavior of suricata to generate the appropriate alert on that rule!

However, there's still a problem when that rule is accompanied by another rule (see new suricata.rules upload):

alert tcp $EXTERNAL_NET any -> $HOME_NET 25 (msg:"VIRUS INBOUND bad file attachment"; flow:to_server,established; content:"content-disposition|3a| attachment|3b|"; nocase; content:".zip|22|"; nocase; within:128; file_data; content:".pdf.exe"; within:64; sid:3000721; rev:1;)
#
alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"ATTACK-RESPONSES directory listing"; flow:established; content:"Volume Serial Number"; sid:3001292; rev:1;)

The attached test-mail-attach.pcap file should generate alerts for both rules, but the first (file_data) rule you wrote the fix for won't trigger unless I comment out the second one, which is the only one generating an alert as long as both are active!

This suggests there's still something wonky going on. Please advise if, instead of reopening this bug, I should open a new bug for this issue, and/or open an issue on github!

Actions #21

Updated by Gabriel Somlo over 3 years ago

Opened new bug #3190 to keep things simple.

Actions #22

Updated by Victor Julien over 3 years ago

  • Related to Bug #3190: file_data inspection inhibited by additional (non-file_data) content match rule added
Actions

Also available in: Atom PDF