Project

General

Profile

Feature #2741

netmap: add support for lb and vale switches

Added by booble tins about 1 month ago. Updated 29 days ago.

Status:
Assigned
Priority:
Normal
Assignee:
Target version:
Effort:
Difficulty:

Description

The documentation suggests that we should be able to use netmap's lb with Suricata: https://suricata.readthedocs.io/en/latest/performance/packet-capture.html#load-balancing

I have lb compiled (FreeBSD 11.2) and moving packets to the netmap pipes eg "netmap:igb{0", but I don't seem to be able to tell Suricata's threads to use those pipes/interfaces to accept the packets or pass them back to the host. I've tried various iterations of:

 - interface: netmap:igb0{0
   copy-iface: igb0+
 - interface: netmap:igb0{0+
   copy-iface: igb0

I've also tried various versions of --netmap=netmap:igb0{0 in the run params.

Initial output from Suricata shows the interface names truncated:

12/12/2018 -- 22:16:54 - <Info> -- Shortening device name to: netm..b0{0
12/12/2018 -- 22:16:54 - <Info> -- Shortening device name to: netm..0{0+

system.log shows:

12/12/2018 -- 22:17:43 - <Info> -- Going to use 1 thread(s)
12/12/2018 -- 22:17:43 - <Error> -- [ERRCODE: SC_ERR_SYSCALL(50)] - Unable to get flags for iface "netmap:igb0{0": Device not configured
12/12/2018 -- 22:17:43 - <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Can not access to interface 'netmap:igb0{0'
12/12/2018 -- 22:17:43 - <Error> -- [ERRCODE: SC_ERR_SYSCALL(50)] - Unable to get caps for iface "netmap:igb0{0": Device not configured

I'm running Suricata 4.0.6 RELEASE in IPS mode with desired processor affinity set. I'm trying to use "worker" mode as opposed to "autofp" per my understanding of the best practices discussed here: https://lists.openinfosecfoundation.org/pipermail/oisf-devel/2013-March/002167.html and here: https://lists.openinfosecfoundation.org/pipermail/oisf-users/2016-March/005829.html

I only see a single reference to using Suricata with netmap pipes on the mailing list, and the author there never got it running (emailed him today): https://lists.openinfosecfoundation.org/pipermail/oisf-users/2017-February/006807.html

Suricata+netmap works, but I haven't found a way to bind it to lb. Is that currently possible?

History

#1 Updated by booble tins about 1 month ago

booble tins wrote:

Suricata+netmap works, but I haven't found a way to bind it to lb. Is that currently possible?

If I am understanding the source correctly, it looks like Suricata is using the low-level native netmap API to open a file descriptor from /dev/netmap rather than netmap_user.h's nm_open() helper function (the "recommended" way):
https://github.com/OISF/suricata/blob/24806c21024e2f799e1cf4d0355e7e5e718224e3/src/source-netmap.c#L229

From the netmap(4) examples:

USING THE NATIVE API
...
fd = open("/dev/netmap", O_RDWR);

So Suricata only has a file descriptor without any of the additional options available in nm_open as described here: https://www.freebsd.org/cgi/man.cgi?query=netmap

The recommended way to bind a file descriptor to a port is to use function nm_open(..) (see LIBRARIES) which parses names to access specific port types and enable features. In the following we document the main features.

I think what that boils down to is that the use of "lb" isn't currently possible despite several references to it in the Suricata docs? To use flow-based load balancing would require access to the pipes which isn't possible?

#2 Updated by Victor Julien about 1 month ago

  • Tracker changed from Support to Feature
  • Subject changed from Support for netmap + lb? to netmap: add support for lb and vale switches
  • Status changed from New to Assigned
  • Assignee set to Victor Julien
  • Target version set to 5.0beta1

Yeah that seems correct. I'm working on a new version that does use the user api and does support lb and vale switches. I should be out for testing soon.

#3 Updated by Victor Julien about 1 month ago

Could you give this pull request a test? https://github.com/OISF/suricata/pull/3596

#5 Updated by booble tins 30 days ago

Steps taken:
  • built FreeBSD 11-STABLE from source
  • built lb from source
  • built Suricata with --enable-netmap from Github Master branch from source with netmap PR
  • modified suricata.yml to enable netmap with 2 threads
  • ran `lb -i em0 -p 2` for 2 pipelines
  • ran `suricata --netmap`

Some notes on the documentation here: https://github.com/OISF/suricata/blob/0518ef8500a7ca14933f1145a63fed1c3c5394a9/doc/userguide/capture-hardware/netmap.rst

  • FreeBSD 11.2-RELEASE comes with netmap built-in, but the lb tool is not bundled (as source or compiled).
  • FreeBSD 11-STABLE contains the recent commits in which lb has been added bundled into the source, but it doesn't compile in a generic build from source and must be compiled manually using `make` from /usr/src/tools/tools/netmap/
  • After compiling `make install` isn't effective and the compiled lb must be copied to eg /usr/local/bin
  • A note on CPU affinity and thread settings with lb+netmap may be useful (I'm still trying to determine if it's working as expected)
  • A note on lb+netmap + alternate copy modes may be useful. Is Suricata handling the bridging for us with copy-mode: ips, or do we need to use the netmap pipe names in the configuration? (Again, still trying to determine how this is working)

There may be better ways of building lb (maybe it's an optional module?), but this was a standard build-from source with GENERIC.

Suricata is up and running with netmap pipes in the VM, I'll see if I can get it on a live firewall for some testing in IPS mode with CPU affinity and report back.


If you forget the startup step `suricata --netmap=netmap:suricata` and use the normal `suricata --netmap` then output looks like below (I mention this because I missed that the first time through, others might as well).

Snip of Suricata output

SC_ERR_NETMAP_CREATE] - opening devname netmap:em0-1 failed: Invalid argument

lb output during Suricata startup

netmap_interp_ringid     invalid ring id 1
nm_open                  NIOCREGIF failed: Invalid argument em0-1/x
netmap_interp_ringid     invalid ring id 1
nm_open                  NIOCREGIF failed: Invalid argument em0-1

#6 Updated by booble tins 29 days ago

Please ignore the startup comment at the end above, I had it backwards.

I'm running into issues getting something like the following configuration to work:

netmap:
 - interface: netmap:em0
   copy-mode: ips
   copy-iface: netmap:em0^
 - interface: netmap:em0^
   copy-mode: ips
   copy-iface: netmap:em0

That configuration gives me:

[100252] 21/12/2018 -- 23:51:47 - (suricata.c:2629) <Info> (PostDeviceFinalizedSetup) -- Netmap: Setting IPS mode
[100252] 21/12/2018 -- 23:51:49 - (runmode-netmap.c:308) <Perf> (ParseNetmapConfig) -- Using 2 threads for interface netmap:em0
[100252] 21/12/2018 -- 23:51:49 - (util-runmodes.c:297) <Info> (RunModeSetLiveCaptureWorkersForDevice) -- Going to use 2 thread(s)
[100254] 21/12/2018 -- 23:51:49 - (source-netmap.c:357) <Notice> (NetmapOpen) -- opened netmap:em0}0 from netmap:em0: 0x8116c9000
** Output from lb here: ** 309.375762 nm_open [608] unexpected character: '}' em0^}0
[100254] 21/12/2018 -- 23:51:49 - (source-netmap.c:348) <Error> (NetmapOpen) -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - opening devname netmap:em0^}0 failed: Invalid argument

Is the caret symbol misplaced there? Should it be trying to open `netmap:em0}0^` ?

Also available in: Atom PDF