Project

General

Profile

Actions

Bug #2555

closed

Ensure strings in eve-log are json-encodable

Added by Maurizio Abba over 5 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
medium
Difficulty:
low
Label:

Description

Halfway between bug and feature.

In our current code any binary data would make function json_string fail. In case of failure, this function return NULL.
It would be cool to add a check before trying to dump data that, in case of binary data being sent, would just hex-encode the result before sending it to json_string.

The idea would be to create a SCJsonString that takes as parameter a const char * and check the return value of json_string, returning it if != NULL.
Otherwise, for every character belonging to the string it would apply the following transformation:
if the character is printable -> keep it as it is
if the character is NOT printable -> transform it in its hex-encoded version.

Note that this transformation would force the encoding of any non json-encodable string to ASCII.

There is one more drawback: for sake of saving dynamic allocation, we will need to hardcode a max boundary on the size of the final buffer after the transformation: the buffer can be limited to 2048 (a string bigger than 2048) would be suspicious. Note that this limitation DOES NOT apply to any string that can't have binary data (such as base64 encoding of TLS certificate).

Attached, an example pcap with a file transfer with binary data in the user agent.
The resulting http.log eve line is

{
  "timestamp": "2015-03-25T15:40:41.558505+0000",
  "flow_id": 149850720655109,
  "pcap_cnt": 7,
  "event_type": "http",
  "src_ip": "192.168.1.44",
  "src_port": 58068,
  "dest_ip": "127.0.0.1",
  "dest_port": 80,
  "proto": "TCP",
  "tx_id": 0,
  "http": {
    "hostname": "ev0lve.org",
    "url": "/a/expl/2008.mp4",
    "http_content_type": "video/mp4",
    "http_method": "GET",
    "protocol": "HTTP/1.1",
    "status": 200,
    "length": 195
  }
}

with the modification it would become

{
  "timestamp": "2015-03-25T15:40:41.558505+0000",
  "flow_id": 67447478114053,
  "pcap_cnt": 7,
  "event_type": "http",
  "src_ip": "192.168.1.44",
  "src_port": 58068,
  "dest_ip": "127.0.0.1",
  "dest_port": 80,
  "proto": "TCP",
  "tx_id": 0,
  "http": {
    "hostname": "ev0lve.org",
    "url": "/a/expl/2008.mp4",
    "http_user_agent": "Wget/1.\\xFF3.4 (linux-gnu)",
    "http_content_type": "video/mp4",
    "http_method": "GET",
    "protocol": "HTTP/1.1",
    "status": 200,
    "length": 195
  }
}

There is also a small change in the rust code. It's actually not definitive, and something where I would like to have some inputs. In the current code the data would just be returned as ASCII, losing any utf-8 encoding capability provided by json.
I think we should change this behavior, but I don't have any good idea other than just calling the SCJsonString function from json.rs deleting the current to_cstring code.

What do you guys think?


Files

2008.mp4-binary-ua.pcap (1.48 KB) 2008.mp4-binary-ua.pcap PCAP showing http user agent with binary data Maurizio Abba, 07/25/2018 03:21 PM
Actions #1

Updated by Maurizio Abba over 5 years ago

  • Status changed from New to Resolved
Actions #2

Updated by Victor Julien over 5 years ago

  • Status changed from Resolved to Closed
  • Target version set to 4.1rc2
Actions

Also available in: Atom PDF