Project

General

Profile

Actions

Bug #4106

closed

Duplicate TLS subjects in tls metadata.

Added by Cooper Nelson about 2 years ago. Updated about 1 month ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
low
Difficulty:
Label:

Description

We are seeing the tls subject duplicated in tls metadata associated with EVE suricata alerts.

The subject is not duplicated in the associated tls event.


Files

eve.jzon.gz (18.8 MB) eve.jzon.gz Phil Rzewski, 04/02/2021 04:40 PM

Subtasks 1 (0 open1 closed)

Bug #5420: Duplicate TLS subjects in tls metadata. (6.0.x backport)ClosedShivani BhardwajActions
Actions #1

Updated by Victor Julien about 2 years ago

Are you able to share a pcap?

Actions #2

Updated by Victor Julien about 2 years ago

  • Status changed from New to Feedback
  • Assignee set to Cooper Nelson
Actions #3

Updated by Phil Rzewski over 1 year ago

I bumped into this problem as well. I've been able to reproduce the problem with my favorite public data source, https://archive.wrccdc.org/pcaps/2018/wrccdc.2018-03-23.010014000000000.pcap.gz.

Here's the steps of me reproducing this on a scratch Ubuntu 20.04 Linux VM.

$ sudo add-apt-repository -y ppa:oisf/suricata-stable
$ sudo apt update
$ sudo apt-get -y --no-install-recommends install suricata
$ sudo suricata-update
$ sudo chmod go+rx /var/lib/suricata/rules
$ wget https://archive.wrccdc.org/pcaps/2018/wrccdc.2018-03-23.010014000000000.pcap.gz
$ gunzip wrccdc.2018-03-23.010014000000000.pcap.gz 

$ suricata -r wrccdc.2018-03-23.010014000000000.pcap 
2/4/2021 -- 16:13:18 - <Notice> - This is Suricata version 6.0.2 RELEASE running in USER mode
2/4/2021 -- 16:13:46 - <Notice> - all 3 packet processing threads, 4 management threads initialized, engine started.
2/4/2021 -- 16:14:05 - <Notice> - Signal Received.  Stopping engine.
2/4/2021 -- 16:14:11 - <Notice> - Pcap-file module read 1 files, 1650919 packets, 473586644 bytes

It seems that jq de-duplicates these silently, but it was flagged by the zq tool from the Zed platform (https://github.com/brimdata/zed).

$ zq eve.json > /dev/null
eve.json: duplicate field subject

I was then able to isolate the first instance of it at line 121 of my eve.json output.

$ head -121 eve.json | tail -1
{"timestamp":"2018-03-23T19:58:22.825466+0000","flow_id":1250013126474967,"pcap_cnt":2065,"event_type":"alert","src_ip":"10.47.2.156","src_port":49285,"dest_ip":"52.85.83.187","dest_port":443,"proto":"TCP","metadata":{"flowbits":["tcp.retransmission.alerted"],"flowints":{"tcp.retransmission.count":10}},"alert":{"action":"allowed","gid":1,"signature_id":2210054,"rev":1,"signature":"SURICATA STREAM excessive retransmissions","category":"Generic Protocol Command Decode","severity":3},"tls":{"subject":"C=US, ST=California, L=Mountain View, O=Mozilla Corporation, OU=Cloud Services, CN=*.services.mozilla.com","issuerdn":"C=US, O=DigiCert Inc, CN=DigiCert SHA2 Secure Server CA","subject":"C=US, ST=California, L=Mountain View, O=Mozilla Corporation, OU=Cloud Services, CN=*.services.mozilla.com","issuerdn":"C=US, O=DigiCert Inc, CN=DigiCert SHA2 Secure Server CA","serial":"01:7E:45:A3:1A:A5:0B:C3:50:53:BC:50:F9:B6:9B:AD","fingerprint":"db:52:69:e4:eb:9a:b5:e0:ce:26:94:0b:df:c4:b0:de:87:c8:4d:f1","sni":"activity-stream-icons.services.mozilla.com","version":"TLS 1.2","notbefore":"2017-10-03T00:00:00","notafter":"2020-01-08T12:00:00","ja3":{"hash":"0ffee3ba8e615ad22535e7f771690a28","string":"771,49195-49199-52393-52392-49196-49200-49162-49161-49171-49172-51-57-47-53-10,0-23-65281-10-11-35-16-5-13,29-23-24-25,0"},"ja3s":{"hash":"76cc3e2d3028143b23ec18e27dbd7ca9","string":"771,49199,0-65281-11-35-5-16"}},"app_proto":"tls","flow":{"pkts_toserver":51,"pkts_toclient":68,"bytes_toserver":4688,"bytes_toclient":79238,"start":"2018-03-23T19:58:22.802007+0000"}}

Putting that through https://jsonlint.com/ and adding emphasis, we see the problem:

{
    "timestamp": "2018-03-23T19:58:22.825466+0000",
    "flow_id": 1250013126474967,
    "pcap_cnt": 2065,
    "event_type": "alert",
    "src_ip": "10.47.2.156",
    "src_port": 49285,
    "dest_ip": "52.85.83.187",
    "dest_port": 443,
    "proto": "TCP",
    "metadata": {
        "flowbits": ["tcp.retransmission.alerted"],
        "flowints": {
            "tcp.retransmission.count": 10
        }
    },
    "alert": {
        "action": "allowed",
        "gid": 1,
        "signature_id": 2210054,
        "rev": 1,
        "signature": "SURICATA STREAM excessive retransmissions",
        "category": "Generic Protocol Command Decode",
        "severity": 3
    },
    "tls": {
--->        "subject": "C=US, ST=California, L=Mountain View, O=Mozilla Corporation, OU=Cloud Services, CN=*.services.mozilla.com",
        "issuerdn": "C=US, O=DigiCert Inc, CN=DigiCert SHA2 Secure Server CA",
--->        "subject": "C=US, ST=California, L=Mountain View, O=Mozilla Corporation, OU=Cloud Services, CN=*.services.mozilla.com",
        "issuerdn": "C=US, O=DigiCert Inc, CN=DigiCert SHA2 Secure Server CA",
        "serial": "01:7E:45:A3:1A:A5:0B:C3:50:53:BC:50:F9:B6:9B:AD",
        "fingerprint": "db:52:69:e4:eb:9a:b5:e0:ce:26:94:0b:df:c4:b0:de:87:c8:4d:f1",
        "sni": "activity-stream-icons.services.mozilla.com",
        "version": "TLS 1.2",
        "notbefore": "2017-10-03T00:00:00",
        "notafter": "2020-01-08T12:00:00",
        "ja3": {
            "hash": "0ffee3ba8e615ad22535e7f771690a28",
            "string": "771,49195-49199-52393-52392-49196-49200-49162-49161-49171-49172-51-57-47-53-10,0-23-65281-10-11-35-16-5-13,29-23-24-25,0" 
        },
        "ja3s": {
            "hash": "76cc3e2d3028143b23ec18e27dbd7ca9",
            "string": "771,49199,0-65281-11-35-5-16" 
        }
    },
    "app_proto": "tls",
    "flow": {
        "pkts_toserver": 51,
        "pkts_toclient": 68,
        "bytes_toserver": 4688,
        "bytes_toclient": 79238,
        "start": "2018-03-23T19:58:22.802007+0000" 
    }
}

I've attached the full EVE JSON output file as well.

Actions #4

Updated by Flo Sfe 11 months ago

I see the same issue on 7.0

Actions #5

Updated by Eric Leblond 11 months ago

  • Assignee changed from Cooper Nelson to Eric Leblond
  • Effort set to low
  • Difficulty set to low
  • Affected Versions 6.0.1, 6.0.3, 6.0.4, 6.0.5 added
  • Label Beginner added
Actions #6

Updated by Eric Leblond 11 months ago

  • Status changed from Feedback to In Review
Actions #7

Updated by Eric Leblond 11 months ago

  • Label Needs backport to 6.0 added

Side note: all alerts on TLS where subject or issuerdn was present would generate a duplicate making the log entry non JSON valid and unusable in some systems.

Actions #8

Updated by Victor Julien 6 months ago

  • Status changed from In Review to Closed
  • Target version set to 7.0.0-beta1
Actions #9

Updated by Shivani Bhardwaj 5 months ago

  • Status changed from Closed to Resolved
Actions #10

Updated by Victor Julien 5 months ago

  • Difficulty deleted (low)
  • Label deleted (Beginner, Needs backport to 6.0)
Actions #11

Updated by Victor Julien 5 months ago

  • Label Needs backport to 5.0 added

Looks like this also applies to 5?

Actions #12

Updated by Victor Julien 4 months ago

  • Label deleted (Needs backport to 5.0)
Actions #13

Updated by Victor Julien about 1 month ago

  • Status changed from Resolved to Closed
Actions

Also available in: Atom PDF