Project

General

Profile

Actions

Bug #6874

open

libhtp appears to stop parsing HTTP client requests mid-pcap - /libhtp::request_uri_not_seen

Added by Tony Robinson 2 months ago. Updated about 2 months ago.

Status:
Assigned
Priority:
High
Target version:
Affected Versions:
Effort:
medium
Difficulty:
Label:

Description

Hello!

My name is Tony, and I work with the Proofpoint Emerging Threats Team. I'd like to report a problem that I've across in Suricata 8.0.0-dev (3/13 build), 7.0.3, and 6.0.0.16. All builds are on x86-64, utilizing the test platform, Dalton (https://github.com/secureworks/dalton).

We got a report from a user for some new rules to cover "AsukaStealer". If you're interested, take a look at the forum post here:

https://community.emergingthreats.net/t/asukastealer-observerstealer-gen/1470

The bottom line is that towards the end of the post, the submitter claims that they had to do content matches that do not utilize the http buffers in order to get their rules to trigger. Here is a link to the any.run sandbox run (and the pcap): https://app.any.run/tasks/8b1ee45a-87de-4fc5-a755-84546a974a44

For those who don't have (or don't want) an any.run account, I've taken the liberty of attaching the pcap (in a dalton job zip file) associated with this issue.

So, to test the submitter's assertions that HTTP buffers weren't working, I took one of the rules they submitted to us, and transformed it a bit:

alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET MALWARE [ANY.RUN] AsukaStealer SQLite Browser Data Exfiltration Attempt"; flow:established,to_server; http.method; content:"POST"; http.content_type; content:"multipart|2f|form-data|3b 20|boundary|3d|"; http.header; content:"X-Config|3a 20|"; fast_pattern; pcre:"/^(?:Google\x5f|Edge\x5f|Steam\x5f|Firefox\x5fF)/R"; content:"COK|0d 0a|"; within:20; http.header_names; content:"|0d 0a|X-Session|0d 0a|"; content:"|0d 0a|X-Info|0d 0a|"; content:"|0d 0a|X-Config|0d 0a|"; reference:md5,ae5537f1a506140ee101ffdf4605fdcc; reference:url,app.any.run/tasks/7a36fb55-3738-4f40-b760-b443689c9edd; reference:url,community.emergingthreats.net/t/asukastealer-observerstealer-gen; classtype:trojan-activity; sid:1; rev:1;)

For reference, here was the original rule (minus a very slight error that would cause it to not trigger):

alert http any any -> any any (msg:"ET MALWARE [ANY.RUN] AsukaStealer Exfiltrates SQLite Browsers Data"; flow:established,to_server; content:"POST"; startswith; content:"Content-Type|3a 20|multipart|2f|form-data|3b 20|boundary="; distance:0; content:"X-Session|3a| "; within:350; content:"X-Info|3a|"; within:45; content:"X-Config|3a| "; within:350; pcre:"/^(Google_COK|Firefox_COK|Edge_COK)\r/R"; content:"X-ID|3a| "; within:22; threshold:type limit, seconds 120, count 1, track by_src; classtype:credential-theft; reference:md5,ae5537f1a506140ee101ffdf4605fdcc; reference:url,app.any.run/tasks/7a36fb55-3738-4f40-b760-b443689c9edd; reference:url,community.emergingthreats.net/t/asukastealer-observerstealer-gen; metadata:malware_family asukastealer, created_at 2024_03_19; sid:1; rev:1;)

The rule I wrote will not trigger on the sandbox pcap. However, there are several existing ET rules that will trigger on that pcap that make use of HTTP buffers. I've noticed that after the HTTP client POST that sends a large PNG file, libhtp starts throwing errors. Here is the output from the Dalton job's HTTP log:

02/09/2024-05:46:12.189215 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:12.989933 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:13.756718 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:14.867041 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:15.067488 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:15.753600 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:16.471367 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:17.826489 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:18.248283 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:20.315563 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:24.411779 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:26.703613 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:26.928567 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:27.087962 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:27.306797 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:27.508658 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:27.678091 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:27.835970 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:28.044538 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:28.235679 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:28.395542 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:28.642545 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:28.904393 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:29.063432 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:29.264992 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:29.467883 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:46:29.725990 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000
02/09/2024-05:45:52.397646 <hostname unknown>[**]/libhtp::request_uri_not_seen[**]<useragent unknown>[**]192.168.100.185:49208 -> 5.42.66.25:3000

I'm not sure why this is happening. The submitted rule does trigger, but it only appears to trigger on the packet thread. I've attached a screenshot the submitter supplied to show how/why they had to use content matches without sticky buffers to get the rule to work. Just as a sanity check I used flowsynth to create a packet capture that would trigger both of the rules above, and I've attached that. I can confirm that both rules will fire on that pcap.

I thought that maybe I was hitting an http client inspection depth configuration problem. The default for Dalton is 100kb for both client and server inspection depth. I upped this to 100mb just to rule it out, and have attached the job zip file for that as well. The rule I created still refuses to trigger. Near as I can tell the HTTP requests in the sandbox pcap are well-formed, there are no irregularities that I have noticed in the pcap, and copying one of the malicious POST requests and using it to create a new packet capture with flowsynth worked just fine. At this point, I'm at a loss.

If there is any other information I can supply to help in finding the root of this problem and/or correcting it, please let me know.


Files

8e29faafe24b76a2.zip (3.05 MB) 8e29faafe24b76a2.zip Dalton run-job containing eve.json, logs, and pcaps for failed alert triggers Tony Robinson, 03/19/2024 05:49 PM
example.png (130 KB) example.png screencap illustrationg the problem, and what was required for the rule to trigger Tony Robinson, 03/19/2024 06:08 PM
e8fb89bebd2311fd.zip (72.9 KB) e8fb89bebd2311fd.zip dalton job submission with client and server http depth set to 100mb to eliminate inspect depth as a root cause Tony Robinson, 03/19/2024 06:13 PM
77bd6252352783b.pcap (1.04 KB) 77bd6252352783b.pcap flowsynth pcap to confirm rules trigger correctly Tony Robinson, 03/19/2024 06:15 PM
Actions #1

Updated by Victor Julien 2 months ago

  • Status changed from New to Assigned
  • Assignee changed from OISF Dev to Philippe Antoine
  • Priority changed from Normal to High
Actions #2

Updated by Philippe Antoine 2 months ago

@Tony Robinson you should also increase request-body-limit in suricata.yaml as the default value is 100 kbytes and the posted PNG is above 1Mbyte

Is this ok ?

Actions #3

Updated by Tony Robinson 2 months ago

Philippe Antoine wrote in #note-2:

@Tony Robinson you should also increase request-body-limit in suricata.yaml as the default value is 100 kbytes and the posted PNG is above 1Mbyte

Is this ok ?

Hey Philippe,

I'm sorry if this wasn't immediately clear, but I indicated in the ticket that I made that change already. Please see the e8fb89bebd2311fd.zip file, and the suricata.yaml contained within, and you'll see that request-body-limit was set to 100mb, and it still resulted in missed alerts. But even so, the HTTP requests after the .PNG file are all separate HTTP requests in the same TCP stream, shouldn't they have their own separate request body byte sizes?

Actions #4

Updated by Philippe Antoine about 2 months ago

Reusing 1267a00ac73f67f9_suricata.yaml I do not get any more request_uri_not_seen by raising also stream.reassembly.depth to 10Mb instead of 1Mb (I thought you were talking about this setting in the first post)

With this, do you get the expected results ?

shouldn't they have their own separate request body byte sizes?

That looks indeed a fair point.

But, I would like first to be sure to have assessed every problem...

Actions

Also available in: Atom PDF