Project

General

Profile

Actions

Task #2167

open

tracking: eve enhancements

Added by Victor Julien over 7 years ago. Updated 3 months ago.

Status:
New
Priority:
Normal
Assignee:
Target version:
Effort:
Difficulty:
Label:

Description

Parts of EVE are not working as well as we want. Changing those would be a breaking change in some cases. Consider a new version of eve that is free to break things.

Some initial notes:

  • config format: the 'types' list should be a map instead
  • config format: much too verbose
  • config format 'eve-log' should just be 'eve'
  • config should have better defaults
  • output of buffers, see #2166
  • output of DNS is too verbose #2086 #1198
  • output HTTP: fix typos like #2000
  • output HTTP: http.http_user_agent is redundant. Could just be http.user_agent

Related issues 6 (5 open1 closed)

Related to Suricata - Bug #2000: eve.http: http_refer should be http_refererRejectedActions
Related to Suricata - Task #6443: Suricon 2023 brainstormAssignedVictor JulienActions
Related to Suricata - Bug #6458: eve/http: discrepancy in http events and http objects logged in alertsNewOISF DevActions
Related to Suricata - Bug #6400: log of DNS answer is in wrong direction NewEric LeblondActions
Related to Suricata - Feature #4853: eve: Add information about Suricata versionNewOISF DevActions
Related to Suricata - Feature #7101: eve: add number of flowbits in protocol records and alertsFeedbackPeter ManevActions
Actions #1

Updated by Victor Julien over 7 years ago

  • Description updated (diff)
Actions #2

Updated by Andreas Herz over 7 years ago

  • Assignee set to OISF Dev
  • Target version set to TBD

Would the old one be deprecated someday in the future or will they coexist for a longtime?

Actions #3

Updated by Victor Julien over 7 years ago

Goal here is first to see what we would like to modify. Then we can see if it can be done within the current eve or that we need a fork. If the latter we'll have to consider how that would work out. But for now I would like to focus on what kind of changes we'd like to make to the input.

Actions #4

Updated by Jason Ish about 7 years ago

I believe we had a discussion on eve versioning, but it must have been online as I can't find any record of it.

On reviewing https://github.com/OISF/suricata/pull/2884, we see https://github.com/OISF/suricata/pull/2884/files#diff-cd6d639bc75c8f4e5201806504a446beR281 which ups the version of "eve" to 2, but only for a DNS change. If we follow this, we'll have to up it to 3 when we make a change to something like TLS for example.

Instead we maybe need 2 levels of version. An eve version for the outer layer of the "base" eve object, then a version for the event type. So using the above PR as an example, only DNS would be upped to version 2. The "dns" object would contain a "version" field of 2.

The configuration file would set the version. The default configuration file always containing the latest version. So on upgrade, if the user does not upgrade their configuration file they stick with the old version - perhaps with deprecation warnings. While I don't expect version updates often, we will want to limit the number of versions being maintained.

This does NOT address the issue of breaking eve log readers that may expect certain fields, or certain fields to be of a certain type that Victor has brought up. But that is not really related to versioning.

Actions #5

Updated by Victor Julien about 7 years ago

Good points Jason, I agree. Just adding here that it requires us to carefully document each version and subversion.

Actions #6

Updated by Jason Ish about 7 years ago

Something else that we might consider is how we name the "list" items under the outputs. Right now we have:

outputs:
  - fast:
      enabled: yes
  - eve-log:
      enabled: yes

This adds one level of extra hierarchy than is probably needed. And means the following is actually valid YAML:
outputs:
  - fast:
      enabled: yes
    eve-log:
      enabled: yes

What might be better is where the log type is set as a type field:

outputs:
  - type: fast
    enabled: yes
  - type: eve-log
    enabled: yes

This one is a bit subjective, but I think the current way adds one more level of complexity and indirection.

Actions #7

Updated by Victor Julien over 5 years ago

  • Related to Bug #2000: eve.http: http_refer should be http_referer added
Actions #8

Updated by Victor Julien over 1 year ago

  • Status changed from New to Assigned
  • Assignee changed from OISF Dev to Jason Ish
Actions #9

Updated by Philippe Antoine about 1 year ago

  • Related to Task #6443: Suricon 2023 brainstorm added
Actions #10

Updated by Philippe Antoine about 1 year ago

Question : do we keep multiple possible (configurable) outputs in the same suricata version cf http.http_user_agent vs http.user_agent
Sascha says that having both may make suricata major version bumping easier for some users

Actions #11

Updated by Victor Julien about 1 year ago

  • Related to Bug #6458: eve/http: discrepancy in http events and http objects logged in alerts added
Actions #12

Updated by Victor Julien 10 months ago

  • Related to Bug #6400: log of DNS answer is in wrong direction added
Actions #13

Updated by Victor Julien 10 months ago

  • Related to Feature #4853: eve: Add information about Suricata version added
Actions #14

Updated by Jason Ish 5 months ago

  • Related to Feature #7101: eve: add number of flowbits in protocol records and alerts added
Actions #15

Updated by Jason Ish 5 months ago

  • Tracker changed from Feature to Task
  • Subject changed from eve-ng to tracking: eve enhancements
  • Assignee changed from Jason Ish to OISF Dev
Actions #16

Updated by Philippe Antoine 3 months ago

  • Status changed from Assigned to New
Actions

Also available in: Atom PDF