Suricata

I've already been looking at this a bit and was going down this path...

includes:
- include1
- include2

These would be top level includes, as if include1 and include2 were inlined into suricata.yaml by a preprocessor. This is similar to the Google app engine includes field in yaml, but not restrictive on what it pulls in.

The other type of include I was thinking of was one that was rooted at a parent. For example:

--- suricata.yaml ---
logging: !include logging.yaml
---

--- logging.yaml ---
default-log-level: notice

outputs:
- console:
enabled: yes
---

Both these types of includes would be processed as they are found, rather than scanning the configuration tree for includes and then adding them in.

I got a chance to look at this today, the result is at https://github.com/jasonish/suricata/tree/config-includes

Syntax is like:

include: somefile
include: otherfile

This will be included into the root configuration node. The affect is as if they were inlined.

The other method is to include a file under an existing configuration node.. For example:

outputs: !include outputs.yaml

outputs.yaml would now look like:

--- outputs.yaml ---

- fast
    enabled: yes
    filename: fast.log
    append: yes

- unified2-alert:
    enabled: yes

Thoughts on the format?

Victor Julien wrote:

When including to update the root configuration, how will it work if things are defined in multiple files?

--- myinclude.yaml ---
[...]

--- suricata.yaml ---
[...]

What will happen?

Especially with lists this may be interesting. Will a list be appended to, or overwritten, or do we error out?

Right now the same thing will happen that would happen if you redefined something in the single file. That is, simple values like mysetting will get overwritten. Lists by default append, but this wasn't by design so the indices reset. I do plan to address this in another bug.

I'm tempted to say that new values override existing values, including lists - this is what pyyaml does. Otherwise we'll have to introduce additional syntax to denote if you want append, or overwrite behaviour which could get complicated and/or confusing.

I do plan to implement YAML anchors and aliases, which might help here..

For example:

my-windows-hosts: &windows-hosts
- 192.168.1.0/24
- 10.0.0.0/8

host-os-policy:
- windows: *windows-hosts

In this case host-os-policy.windows is pointer to the value in my-windows-hosts.

Jason Ish wrote:

I'm tempted to say that new values override existing values, including lists - this is what pyyaml does. Otherwise we'll have to introduce additional syntax to denote if you want append, or overwrite behaviour which could get complicated and/or confusing.

I agree, lets overwrite. I think we should add a warning message if we do though.

I do plan to implement YAML anchors and aliases, which might help here..

For example:

my-windows-hosts: &windows-hosts
- 192.168.1.0/24
- 10.0.0.0/8

host-os-policy:
- windows: *windows-hosts

In this case host-os-policy.windows is pointer to the value in my-windows-hosts.

I like this, although is it 'official' yaml? Will 3rd party yaml parsers know what this is?

Btw, this anchors and aliases support should have it's own ticket I think.

Victor Julien wrote:

Jason Ish wrote:

I'm tempted to say that new values override existing values, including lists - this is what pyyaml does. Otherwise we'll have to introduce additional syntax to denote if you want append, or overwrite behaviour which could get complicated and/or confusing.

I agree, lets overwrite. I think we should add a warning message if we do though.

Ok, I'll do that as part of this ticket, as file includes may bring the question up.

I do plan to implement YAML anchors and aliases, which might help here..

For example:

my-windows-hosts: &windows-hosts
- 192.168.1.0/24
- 10.0.0.0/8

host-os-policy:
- windows: *windows-hosts

In this case host-os-policy.windows is pointer to the value in my-windows-hosts.

I like this, although is it 'official' yaml? Will 3rd party yaml parsers know what this is?

Btw, this anchors and aliases support should have it's own ticket I think.

Yes, this should be its own ticket.

Anchors and aliases are official YAML.
- http://yaml.org/spec/1.1/#id863390
There are examples within that doc.

A basic yaml parser like libyaml used in Suricata doesn't do much with these. It just stores them as attributes, and we have to link up the alias to the anchor. But higher level parsers, like PyYAML and SnakeYaml (Java) resolve the aliases as expected.

The only problem that I see with anchors and aliases is they are lost upon parsing, at least with PyYAML. So if you parse the yaml, then are planning to emit the yaml in an effort to clean it up perhaps, you will loose the aliases and anchors, or at least have them mangled in some way where they lose their original names.

Jason Ish wrote:

I like this, although is it 'official' yaml? Will 3rd party yaml parsers know what this is?

Btw, this anchors and aliases support should have it's own ticket I think.

Yes, this should be its own ticket.

Anchors and aliases are official YAML.
- http://yaml.org/spec/1.1/#id863390
There are examples within that doc.

A basic yaml parser like libyaml used in Suricata doesn't do much with these. It just stores them as attributes, and we have to link up the alias to the anchor. But higher level parsers, like PyYAML and SnakeYaml (Java) resolve the aliases as expected.

The only problem that I see with anchors and aliases is they are lost upon parsing, at least with PyYAML. So if you parse the yaml, then are planning to emit the yaml in an effort to clean it up perhaps, you will loose the aliases and anchors, or at least have them mangled in some way where they lose their original names.

That is a disadvantage we have with the comments as well. I guess the case where ppl use both tools and manual edits isn't really feasible.