Project

General

Profile

Actions

Support #2512

closed

http events - Weird unicode characters and truncation in some of http_method/http_user_agent fields

Added by Gonzalez Marc almost 6 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Affected Versions:
Label:

Description

Hello,

We have recently upgraded our ids in production from suricata version 3.2.1 to version 4.0.4 via the package manager(ppa repository) on Ubunutu 14.04.5 LTS.

Since the upgrade, we have noticed in the http event logs weird unicode characters in the values of the http_method and http_user_agent fields and random string truncation.
We kept the suricata.yml from version 3.2.1 and did not make any change in http log setup.
This problem has also been noticed in our test ids that has a low load(system and bandwith) but was more rare due too low traffic(only a few events).


Problematic User Agent sample:

{"timestamp":"2018-06-04T05:40:49.313734+0000","flow_id":192722981970376,"in_iface":"bond0","event_type":"http","src_ip":"x.x.x.x","src_port":56217,
"dest_ip":"x.x.x.x","dest_port":8080,"proto":"TCP","tx_id":0,"http":{"hostname":"www.bing.com","url":"www.bing.com:443",
*"http_user_agent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64; ServiceUI 11) AppleWebKit\/537.36 (KHTML, like Gecko) 
Chrome\/58.0.3029.110 Safari\/537.36 Edge\/16.16299"*,"pragma":"no-cache","http_method":"CONNECT","protocol":"HTTP\/1.0","length":8302}}
*=> the event is not problematci, it has a full ua string and the same flow id as the second event*

{"timestamp":"2018-06-04T05:40:49.313734+0000","flow_id":192722981970376,"in_iface":"bond0","event_type":"http","src_ip":"x.x.x.x","src_port":56217,
"dest_ip":"x.x.x.x","dest_port":8080,"proto":"TCP","tx_id":1,"http":{*"http_user_agent":"Mozi\u0017\u0003\u0003"*,
"protocol":"www.bing.com:443 HTTP\/1.0","length":0}}

*=> this seconde event has atruncated ua string with unicode characters and hte same flow id as the first event* 

{"timestamp":"2018-05-30T05:23:44.577489+0000","flow_id":1546874169271876,"in_iface":"bond0","event_type":"http","src_ip":"x.x.x.x",
"src_port":17561,"dest_ip":"x.x.x.x","dest_port":8080,"proto":"TCP","tx_id":0,"http":{"url":"pool-adhese.com:443","http_user_agent":"Mozilla\/5.0\u0016\u0003\u0003" 
,"http_method":"CONNECT","protocol":"HTTP\/1.1","length":0}}

Problematic http_method sample

{"timestamp":"2018-06-03T08:02:10.003933+0000","flow_id":961959389296609,"event_type":"http","src_ip":"x.x.x.x",
"src_port":53722,"dest_ip":"x.x.x.x","dest_port":8080,"proto":"TCP","tx_id":1,"http":{"http_method":"\u0016\u0003\u0003\\0F\u0010\\0\\0BA\u0004H\u0002","length":0}}

Apt history
Commandline: apt-get install --only-upgrade suricata
Install: libhtp2:amd64 (0.5.26-2ubuntu4, automatic)
Upgrade: suricata:amd64 (3.2.1-0ubuntu1, 4.0.4-2ubuntu4)

Packages upgraded:
ii  libhtp1                             0.5.x.201707130636~ubuntu14.04.1    amd64        HTTP normalizer and parser library
ii  libhtp2                             1:0.5.26-2ubuntu4                   amd64        HTTP normalizer and parser library => installed with version 4.0.4
ii  suricata                            4.0.4-2ubuntu4                      amd64        Suricata open source multi-thread IDS/IPS/NSM system.

http eve log setup suricata.yml(Not changed)

- eve-log:
enabled: yes
filetype: regular #regular|syslog|unix_dgram|unix_stream
filename: eve-http.json
types:
- http:
extended: yes # enable this for extended logging information # custom allows additional http fields to be included in eve-log # the example below adds three additional fields when uncommented
#custom: [Accept-Encoding, Accept-Language, Authorization]
custom: [accept, accept-charset, accept-encoding, accept-language,
accept-datetime, authorization, cache-control, set-cookie, cookie, from,
max-forwards, origin, pragma, proxy-authorization, proxy-connection, range, te, via,
x-requested-with, dnt, x-forwarded-proto, x-requested-with, accept-range, age,
allow, connection, content-encoding, content-language,
content-length, content-location, content-md5, content-range,
content-type, date, etags, last-modified, link, location,
proxy-authenticate, referrer, refresh, retry-after, server,
set-cookie, trailer, transfer-encoding, upgrade, vary, warning,
www-authenticate]

Could you help us to investigate that issue?

Thanks in advance,


Files

sample_10.62.112.41_62227.pcap (10.1 KB) sample_10.62.112.41_62227.pcap Gonzalez Marc, 06/08/2018 02:34 PM
Actions #1

Updated by Gonzalez Marc almost 6 years ago

Hi,

Further test:

I installed and tested the previous version 4.0.3 (packages libhtp-0.5.25-1_4.0.3-1ubuntu2_amd64.deb and suricata_4.0.3-1ubuntu2_amd64.deb) and I got the same problem.

Reinstalling back version 3.2.1 (packages libhtp1_0.5.x.201701112247~ubuntu14.04.1_amd64.deb and suricata_3.2.1-0ubuntu1_amd64.deb) fixed the problem.

Actions #2

Updated by Jason Ish almost 6 years ago

Gonzalez Marc wrote:

Hi,

Further test:

I installed and tested the previous version 4.0.3 (packages libhtp-0.5.25-1_4.0.3-1ubuntu2_amd64.deb and suricata_4.0.3-1ubuntu2_amd64.deb) and I got the same problem.

Reinstalling back version 3.2.1 (packages libhtp1_0.5.x.201701112247~ubuntu14.04.1_amd64.deb and suricata_3.2.1-0ubuntu1_amd64.deb) fixed the problem.

By any chance can you share a pcap file that exhibits this issue? Privately if needed.

Thanks.

Actions #3

Updated by Gonzalez Marc almost 6 years ago

Hello,

You can find in attachment a small pcap with a problematic packet.

Offline tests made with that pcap file

http event log output:

egrep 10.62.112.41.*62227 logs_suricata_3.2.1_pcap_replay/eve-http.json


{"timestamp":"2018-06-07T13:30:29.742850+0000","flow_id":418474559356980,"pcap_cnt":813833,"event_type":"http","src_ip":"10.62.112.41","src_port":62227,
"dest_ip":"10.7.108.210","dest_port":8080,"proto":"TCP","tx_id":0,"http":{"hostname":"secure-assets.rubiconproject.com","url":"secure-assets.rubiconproject.com:443",
"http_user_agent":"Mozilla\/5.0 (Windows NT 10.0; WOW64; Trident\/7.0; rv:11.0) like Gecko" 
,"pragma":"no-cache","http_method":"CONNECT","protocol":"HTTP\/1.0","length":0}}

egrep 10.62.112.41.*62227 logs_suricata_4.0.3_pcap_replay/eve-http.json


{"timestamp":"2018-06-07T13:30:29.856419+0000","flow_id":1356209801478196,"event_type":"http","src_ip":"10.62.112.41","src_port":62227,"dest_ip":"10.7.108.210","dest_port":8080,
"proto":"TCP","tx_id":0,
"http":{"hostname":"secure-assets.rubiconproject.com","url":"secure-assets.rubiconproject.com:443",
"http_user_agent":"Mozilla\/5.0 (Windows NT 10.0; WOW64; Trident\/7.0; rv:11.0) like Gecko" 
,"pragma":"no-cache","http_method":"CONNECT","protocol":"HTTP\/1.0","status":0,"length":1511}}

{"timestamp":"2018-06-07T13:30:29.856419+0000","flow_id":1356209801478196,"event_type":"http","src_ip":"10.62.112.41","src_port":62227,"dest_ip":"10.7.108.210","dest_port":8080,
"proto":"TCP","tx_id":1,"http":{"http_method":"\u0016\u0003\u0003\\0F\u0010\\0\\0BA\u00049s","length":0}}

egrep 10.62.112.41.*62227 logs_suricata_4.0.4_pcap_replay/eve-http.json

{"timestamp":"2018-06-07T13:30:30.923441+0000","flow_id":139168541130804,"event_type":"http","src_ip":"10.62.112.41","src_port":62227,"dest_ip":"10.7.108.210","dest_port":8080,
"proto":"TCP","tx_id":0,"http":{"hostname":"secure-assets.rubiconproject.com","url":"secure-assets.rubiconproject.com:443",
"http_user_agent":"Mozilla\/5.0 (Windows NT 10.0; WOW64; Trident\/7.0; rv:11.0) like Gecko",
"pragma":"no-cache","http_method":"CONNECT","protocol":"HTTP\/1.0","length":3033}}

{"timestamp":"2018-06-07T13:30:30.923441+0000","flow_id":139168541130804,"event_type":"http","src_ip":"10.62.112.41","src_port":62227,"dest_ip":"10.7.108.210","dest_port":8080,
"proto":"TCP","tx_id":1,"http":{"http_method":"\u0016\u0003\u0003\\0F\u0010\\0\\0BA\u00049s","length":0}}
Actions #4

Updated by Gonzalez Marc almost 6 years ago

Futher tests made with the pcap provided:

- Suricata 4.0.4 on Ubuntu 16.04 LTS => Same problem.
- Suricata 4.1 beta1 on Ubuntu 16.04 LTS and 14.04 LTS => No more problem (same output as suricata 3.2.1).

Actions #5

Updated by Peter Manev almost 6 years ago

Thank you for the feedback.
Just to confirm something for the test that you did (quoted bellow)


- Suricata 4.0.4 on Ubuntu 16.04 LTS => Same problem.
- Suricata 4.1 beta1 on Ubuntu 16.04 LTS and 14.04 LTS => No more problem (same output as suricata 3.2.1).

Was that on the same machine(in the case of Ubuntu 16.04 LTS) - aka only the Suricata and libhtp package changed? (everything else including the libjansson is the same)

Actions #6

Updated by Andreas Herz almost 6 years ago

  • Assignee set to OISF Dev
  • Target version set to TBD
Actions #7

Updated by Gonzalez Marc almost 6 years ago

Hi,

Sorry for late reply.

Regarding Ubuntu 16.04, I installed suricata 4.0.4 and 4.1-beta from oisf repositories on the same box (installed and tested one at a time with the pcap provided). Same libhtp2 package version has been used by those two version of suricata.

Version 4.0.4 has the "unicode" problem mentioned above but version 4.1-beta does not seem to have it.

ii libhtp2 1:0.5.26-2ubuntu3 amd64 HTTP normalizer and parser library
ii suricata 4.1.0~beta1-1ubuntu0 amd64 Suricata open source multi-thread IDS/IPS/NSM system.

ii libhtp2 1:0.5.26-2ubuntu3 amd64 HTTP normalizer and parser library
ii suricata 4.0.4-2ubuntu3 amd64 Suricata open source multi-thread IDS/IPS/NSM system.

Actions #8

Updated by Victor Julien over 5 years ago

  • Status changed from New to Assigned
  • Assignee changed from OISF Dev to Victor Julien
  • Priority changed from High to Normal
  • Target version changed from TBD to 4.0.6
  • Affected Versions 4.0.5 added
Actions #9

Updated by Victor Julien over 5 years ago

  • Status changed from Assigned to Closed
Actions

Also available in: Atom PDF