Project

General

Profile

Actions

Bug #4106

closed

Duplicate TLS subjects in tls metadata.

Added by Cooper Nelson over 3 years ago. Updated over 1 year ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
low
Difficulty:
Label:

Description

We are seeing the tls subject duplicated in tls metadata associated with EVE suricata alerts.

The subject is not duplicated in the associated tls event.


Files

eve.jzon.gz (18.8 MB) eve.jzon.gz Phil Rzewski, 04/02/2021 04:40 PM

Subtasks 1 (0 open1 closed)

Bug #5420: Duplicate TLS subjects in tls metadata. (6.0.x backport)ClosedShivani BhardwajActions
Actions #1

Updated by Victor Julien over 3 years ago

Are you able to share a pcap?

Actions #2

Updated by Victor Julien over 3 years ago

  • Status changed from New to Feedback
  • Assignee set to Cooper Nelson
Actions #3

Updated by Phil Rzewski almost 3 years ago

I bumped into this problem as well. I've been able to reproduce the problem with my favorite public data source, https://archive.wrccdc.org/pcaps/2018/wrccdc.2018-03-23.010014000000000.pcap.gz.

Here's the steps of me reproducing this on a scratch Ubuntu 20.04 Linux VM.

$ sudo add-apt-repository -y ppa:oisf/suricata-stable
$ sudo apt update
$ sudo apt-get -y --no-install-recommends install suricata
$ sudo suricata-update
$ sudo chmod go+rx /var/lib/suricata/rules
$ wget https://archive.wrccdc.org/pcaps/2018/wrccdc.2018-03-23.010014000000000.pcap.gz
$ gunzip wrccdc.2018-03-23.010014000000000.pcap.gz 

$ suricata -r wrccdc.2018-03-23.010014000000000.pcap 
2/4/2021 -- 16:13:18 - <Notice> - This is Suricata version 6.0.2 RELEASE running in USER mode
2/4/2021 -- 16:13:46 - <Notice> - all 3 packet processing threads, 4 management threads initialized, engine started.
2/4/2021 -- 16:14:05 - <Notice> - Signal Received.  Stopping engine.
2/4/2021 -- 16:14:11 - <Notice> - Pcap-file module read 1 files, 1650919 packets, 473586644 bytes

It seems that jq de-duplicates these silently, but it was flagged by the zq tool from the Zed platform (https://github.com/brimdata/zed).

$ zq eve.json > /dev/null
eve.json: duplicate field subject

I was then able to isolate the first instance of it at line 121 of my eve.json output.

$ head -121 eve.json | tail -1
{"timestamp":"2018-03-23T19:58:22.825466+0000","flow_id":1250013126474967,"pcap_cnt":2065,"event_type":"alert","src_ip":"10.47.2.156","src_port":49285,"dest_ip":"52.85.83.187","dest_port":443,"proto":"TCP","metadata":{"flowbits":["tcp.retransmission.alerted"],"flowints":{"tcp.retransmission.count":10}},"alert":{"action":"allowed","gid":1,"signature_id":2210054,"rev":1,"signature":"SURICATA STREAM excessive retransmissions","category":"Generic Protocol Command Decode","severity":3},"tls":{"subject":"C=US, ST=California, L=Mountain View, O=Mozilla Corporation, OU=Cloud Services, CN=*.services.mozilla.com","issuerdn":"C=US, O=DigiCert Inc, CN=DigiCert SHA2 Secure Server CA","subject":"C=US, ST=California, L=Mountain View, O=Mozilla Corporation, OU=Cloud Services, CN=*.services.mozilla.com","issuerdn":"C=US, O=DigiCert Inc, CN=DigiCert SHA2 Secure Server CA","serial":"01:7E:45:A3:1A:A5:0B:C3:50:53:BC:50:F9:B6:9B:AD","fingerprint":"db:52:69:e4:eb:9a:b5:e0:ce:26:94:0b:df:c4:b0:de:87:c8:4d:f1","sni":"activity-stream-icons.services.mozilla.com","version":"TLS 1.2","notbefore":"2017-10-03T00:00:00","notafter":"2020-01-08T12:00:00","ja3":{"hash":"0ffee3ba8e615ad22535e7f771690a28","string":"771,49195-49199-52393-52392-49196-49200-49162-49161-49171-49172-51-57-47-53-10,0-23-65281-10-11-35-16-5-13,29-23-24-25,0"},"ja3s":{"hash":"76cc3e2d3028143b23ec18e27dbd7ca9","string":"771,49199,0-65281-11-35-5-16"}},"app_proto":"tls","flow":{"pkts_toserver":51,"pkts_toclient":68,"bytes_toserver":4688,"bytes_toclient":79238,"start":"2018-03-23T19:58:22.802007+0000"}}

Putting that through https://jsonlint.com/ and adding emphasis, we see the problem:

{
    "timestamp": "2018-03-23T19:58:22.825466+0000",
    "flow_id": 1250013126474967,
    "pcap_cnt": 2065,
    "event_type": "alert",
    "src_ip": "10.47.2.156",
    "src_port": 49285,
    "dest_ip": "52.85.83.187",
    "dest_port": 443,
    "proto": "TCP",
    "metadata": {
        "flowbits": ["tcp.retransmission.alerted"],
        "flowints": {
            "tcp.retransmission.count": 10
        }
    },
    "alert": {
        "action": "allowed",
        "gid": 1,
        "signature_id": 2210054,
        "rev": 1,
        "signature": "SURICATA STREAM excessive retransmissions",
        "category": "Generic Protocol Command Decode",
        "severity": 3
    },
    "tls": {
--->        "subject": "C=US, ST=California, L=Mountain View, O=Mozilla Corporation, OU=Cloud Services, CN=*.services.mozilla.com",
        "issuerdn": "C=US, O=DigiCert Inc, CN=DigiCert SHA2 Secure Server CA",
--->        "subject": "C=US, ST=California, L=Mountain View, O=Mozilla Corporation, OU=Cloud Services, CN=*.services.mozilla.com",
        "issuerdn": "C=US, O=DigiCert Inc, CN=DigiCert SHA2 Secure Server CA",
        "serial": "01:7E:45:A3:1A:A5:0B:C3:50:53:BC:50:F9:B6:9B:AD",
        "fingerprint": "db:52:69:e4:eb:9a:b5:e0:ce:26:94:0b:df:c4:b0:de:87:c8:4d:f1",
        "sni": "activity-stream-icons.services.mozilla.com",
        "version": "TLS 1.2",
        "notbefore": "2017-10-03T00:00:00",
        "notafter": "2020-01-08T12:00:00",
        "ja3": {
            "hash": "0ffee3ba8e615ad22535e7f771690a28",
            "string": "771,49195-49199-52393-52392-49196-49200-49162-49161-49171-49172-51-57-47-53-10,0-23-65281-10-11-35-16-5-13,29-23-24-25,0" 
        },
        "ja3s": {
            "hash": "76cc3e2d3028143b23ec18e27dbd7ca9",
            "string": "771,49199,0-65281-11-35-5-16" 
        }
    },
    "app_proto": "tls",
    "flow": {
        "pkts_toserver": 51,
        "pkts_toclient": 68,
        "bytes_toserver": 4688,
        "bytes_toclient": 79238,
        "start": "2018-03-23T19:58:22.802007+0000" 
    }
}

I've attached the full EVE JSON output file as well.

Actions #4

Updated by Flo Sfe about 2 years ago

I see the same issue on 7.0

Actions #5

Updated by Eric Leblond about 2 years ago

  • Assignee changed from Cooper Nelson to Eric Leblond
  • Effort set to low
  • Difficulty set to low
  • Affected Versions 6.0.1, 6.0.3, 6.0.4, 6.0.5 added
  • Label Beginner added
Actions #6

Updated by Eric Leblond about 2 years ago

  • Status changed from Feedback to In Review
Actions #7

Updated by Eric Leblond about 2 years ago

  • Label Needs backport to 6.0 added

Side note: all alerts on TLS where subject or issuerdn was present would generate a duplicate making the log entry non JSON valid and unusable in some systems.

Actions #8

Updated by Victor Julien almost 2 years ago

  • Status changed from In Review to Closed
  • Target version set to 7.0.0-beta1
Actions #9

Updated by Shivani Bhardwaj over 1 year ago

  • Status changed from Closed to Resolved
Actions #10

Updated by Victor Julien over 1 year ago

  • Difficulty deleted (low)
  • Label deleted (Beginner, Needs backport to 6.0)
Actions #11

Updated by Victor Julien over 1 year ago

  • Label Needs backport to 5.0 added

Looks like this also applies to 5?

Actions #12

Updated by Victor Julien over 1 year ago

  • Label deleted (Needs backport to 5.0)
Actions #13

Updated by Victor Julien over 1 year ago

  • Status changed from Resolved to Closed
Actions

Also available in: Atom PDF