Feature #1009: Yaml file inclusion support - Suricata - Open Information Security Foundation

Custom queries

Good First Issues
OISF community

Actions

Copy link

Feature #1009

closed

Yaml file inclusion support

Added by Anoop Saldanha over 10 years ago. Updated over 10 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Jason Ish

Target version:

2.0beta2

Effort:

Difficulty:

Label:

Description

We are probably moving in the direction of splitting our yaml file. It would be
be nice to verify the file inclusion feature and provide support for some of the
below features.

Couple of things that can be probably be discussed -

1. Provide support for specifying multiple yaml file on the command line.

2. If we have a yaml file as -

=Full.yaml=

one:
  two:
    three:
      four: "five_1" 
      five: "five_2
    six:
      seven: "seven"

Should we support a format such as -

=main.yaml=

one:
  two:
    #include "three_v1.yaml" 
    six:
      seven: "seven"

=three_v1.yaml=

three:
  four: "five_1" 
  five: "five_2

=main.yaml=

one:
  two:
    six:
      seven: "seven" 
#include "three_v2.yaml"

=three_v2.yaml=

one:
  two:
    three:
      four: "five_1" 
      five: "five_2

Basically what solution one(three_v1.yaml) does is, it doesn't require you to
specify the parent nodes for the config that you want to include. The node
under which you include the yaml file would be the parent for the included
config.

History
Notes
Property changes

Actions

Copy link

Updated by Jason Ish over 10 years ago

I've already been looking at this a bit and was going down this path...

includes:
- include1
- include2

These would be top level includes, as if include1 and include2 were inlined into suricata.yaml by a preprocessor. This is similar to the Google app engine includes field in yaml, but not restrictive on what it pulls in.

The other type of include I was thinking of was one that was rooted at a parent. For example:

--- suricata.yaml ---
logging: !include logging.yaml
---

--- logging.yaml ---
default-log-level: notice

outputs:
- console:
enabled: yes
---

Both these types of includes would be processed as they are found, rather than scanning the configuration tree for includes and then adding them in.

Actions

Copy link

Updated by Jason Ish over 10 years ago

I got a chance to look at this today, the result is at https://github.com/jasonish/suricata/tree/config-includes

Syntax is like:

include: somefile
include: otherfile

This will be included into the root configuration node. The affect is as if they were inlined.

The other method is to include a file under an existing configuration node.. For example:

outputs: !include outputs.yaml

outputs.yaml would now look like:

--- outputs.yaml ---

- fast
    enabled: yes
    filename: fast.log
    append: yes

- unified2-alert:
    enabled: yes

Thoughts on the format?

Actions

Copy link

Updated by Victor Julien over 10 years ago

Status changed from New to Assigned
Assignee set to Jason Ish
Target version changed from TBD to 2.0rc1

When including to update the root configuration, how will it work if things are defined in multiple files?

--- myinclude.yaml ---

mysetting: 1

--- suricata.yaml ---

include: myinclude.yaml
mysetting: 2

What will happen?

Especially with lists this may be interesting. Will a list be appended to, or overwritten, or do we error out?

Actions

Copy link

Updated by Jason Ish over 10 years ago

Victor Julien wrote:

When including to update the root configuration, how will it work if things are defined in multiple files?

--- myinclude.yaml ---
[...]

--- suricata.yaml ---
[...]

What will happen?

Especially with lists this may be interesting. Will a list be appended to, or overwritten, or do we error out?

Right now the same thing will happen that would happen if you redefined something in the single file. That is, simple values like mysetting will get overwritten. Lists by default append, but this wasn't by design so the indices reset. I do plan to address this in another bug.

I'm tempted to say that new values override existing values, including lists - this is what pyyaml does. Otherwise we'll have to introduce additional syntax to denote if you want append, or overwrite behaviour which could get complicated and/or confusing.

I do plan to implement YAML anchors and aliases, which might help here..

For example:

my-windows-hosts: &windows-hosts
- 192.168.1.0/24
- 10.0.0.0/8

host-os-policy:
- windows: *windows-hosts

In this case host-os-policy.windows is pointer to the value in my-windows-hosts.

Actions

Copy link

Updated by Victor Julien over 10 years ago

Jason Ish wrote:

I'm tempted to say that new values override existing values, including lists - this is what pyyaml does. Otherwise we'll have to introduce additional syntax to denote if you want append, or overwrite behaviour which could get complicated and/or confusing.

I agree, lets overwrite. I think we should add a warning message if we do though.

I do plan to implement YAML anchors and aliases, which might help here..

For example:

my-windows-hosts: &windows-hosts
- 192.168.1.0/24
- 10.0.0.0/8

host-os-policy:
- windows: *windows-hosts

In this case host-os-policy.windows is pointer to the value in my-windows-hosts.

I like this, although is it 'official' yaml? Will 3rd party yaml parsers know what this is?

Btw, this anchors and aliases support should have it's own ticket I think.

Actions

Copy link

Updated by Jason Ish over 10 years ago

Victor Julien wrote:

Jason Ish wrote:

I'm tempted to say that new values override existing values, including lists - this is what pyyaml does. Otherwise we'll have to introduce additional syntax to denote if you want append, or overwrite behaviour which could get complicated and/or confusing.

I agree, lets overwrite. I think we should add a warning message if we do though.

Ok, I'll do that as part of this ticket, as file includes may bring the question up.

I do plan to implement YAML anchors and aliases, which might help here..

For example:

my-windows-hosts: &windows-hosts
- 192.168.1.0/24
- 10.0.0.0/8

host-os-policy:
- windows: *windows-hosts

In this case host-os-policy.windows is pointer to the value in my-windows-hosts.

I like this, although is it 'official' yaml? Will 3rd party yaml parsers know what this is?

Btw, this anchors and aliases support should have it's own ticket I think.

Yes, this should be its own ticket.

Anchors and aliases are official YAML.
- http://yaml.org/spec/1.1/#id863390
There are examples within that doc.

A basic yaml parser like libyaml used in Suricata doesn't do much with these. It just stores them as attributes, and we have to link up the alias to the anchor. But higher level parsers, like PyYAML and SnakeYaml (Java) resolve the aliases as expected.

The only problem that I see with anchors and aliases is they are lost upon parsing, at least with PyYAML. So if you parse the yaml, then are planning to emit the yaml in an effort to clean it up perhaps, you will loose the aliases and anchors, or at least have them mangled in some way where they lose their original names.

Actions

Copy link

Updated by Victor Julien over 10 years ago

Target version changed from 2.0rc1 to 2.0beta2

Actions

Copy link

Updated by Victor Julien over 10 years ago

Jason Ish wrote:

I like this, although is it 'official' yaml? Will 3rd party yaml parsers know what this is?

Btw, this anchors and aliases support should have it's own ticket I think.

Yes, this should be its own ticket.

Anchors and aliases are official YAML.
- http://yaml.org/spec/1.1/#id863390
There are examples within that doc.

A basic yaml parser like libyaml used in Suricata doesn't do much with these. It just stores them as attributes, and we have to link up the alias to the anchor. But higher level parsers, like PyYAML and SnakeYaml (Java) resolve the aliases as expected.

The only problem that I see with anchors and aliases is they are lost upon parsing, at least with PyYAML. So if you parse the yaml, then are planning to emit the yaml in an effort to clean it up perhaps, you will loose the aliases and anchors, or at least have them mangled in some way where they lose their original names.

That is a disadvantage we have with the comments as well. I guess the case where ppl use both tools and manual edits isn't really feasible.

Actions

Copy link