Suricata

Can you share your steps on how you are crosscompiling? I'd like to see if it can be added to CI.

Andreas Herz wrote:

A bit OT, but should this result in an official OpenWRT package?

As far as I can tell, Suricata isn't a OpenWrt package. I found a starting point for making the package and fixed it to work (aside from the file check on cross compile). My build machine is a x64 Ubuntu, but the target is mips64.

If I can get it to work, I'll package it as it's own git and figure out how to submit/PR it. They support Snort, of course, but I need options in my builds (also an officially unsupported device)

Andreas Herz wrote:

A bit OT, but should this result in an official OpenWRT package?

I got Suricata to compile and build into OpenWrt. I had to use --disable-suricata-update and --disable-rust because of the CrossCompile file checks outlined above. Compiled, installed, but reports:

root@OpenWrt:/etc/suricata# suricata -c /etc/suricata/suricata.yaml -s /etc/suricata/rules/signatures.rules -i eth0 -vv
8/7/2019 -- 07:12:03 - <Notice> - This is Suricata version 4.1.4 RELEASE
8/7/2019 -- 07:12:03 - <Info> - CPUs/cores online: 2
8/7/2019 -- 07:12:03 - <Error> - [ERRCODE: SC_ERR_MAGIC_LOAD(197)] - magic_load failed: could not find any valid magic files!

This is outside the scope of the bug report, but I figured I'd let you all know progress was made and how I got around it.

I do have libmagic installed on the device.

root@OpenWrt:/usr/lib# ls *mag
lrwxrwxrwx    1 root     root            17 Jan  1  1970 libmagic.so.1 -> libmagic.so.1.0.0
-rwxr-xr-x    1 root     root        145424 Jul  2 22:32 libmagic.so.1.0.0

OpenWrt looks interested in making Suricata a package, but the upcoming changes to 5.x in the Rust/Cargo requirements is going to cause issues. If I can, I'll see about trying to make an accompanying rust-lang and cargo package for it, but I'd like to attempt that only after I get 4.1.4 working as intended.

Can you add the /etc/suricata/suricata.yaml I guess that's just a config setting issue.

And yes the rust part might be more difficult but since we will make it mandatory it would be recommended.

Can you also paste the suricata --build-info output?

It's a bit odd that you didn't get any warnings related to that if there is no libmagic installed. But if you don't need it you can run --disable-libmagic at the configure step.

I see, but that's a non suricata issue then and needs to be fixed on OpenWRT side.

Donald Hoskins wrote:

Andreas Herz wrote:

Can you also paste the suricata --build-info output?

It's a bit odd that you didn't get any warnings related to that if there is no libmagic installed. But if you don't need it you can run --disable-libmagic at the configure step.

Well, as I said above, I appear to have libmagic installed. It builds, and is present in /usr/lib, but it didn't seem to generate any .mgc file.

Attaching suricata --build-info output

Side note: I'm sure it won't effect anything, but when looking at the --build-info, I noticed I hadn't re-added --enable-pie, so I ran off another compile with it. So, it's got PIE support now..

The full configure arguments are below:

CONFIGURE_ARGS = \
        --prefix="/usr" \
        --sysconfdir="/etc" \
        --localstatedir=/var \
        --disable-gccmarch-native \
        --enable-lua \
        --enable-geopip \
        --enable-nfqueue \
        --enable-gccprotect \
        --enable-debug \
        --enable-pie \
        --disable-suricata-update \
        --disable-rust \
        --host=$(ARCH)

Andreas Herz wrote:

I see, but that's a non suricata issue then and needs to be fixed on OpenWRT side.

Yes. I wanted to ensure that it wasn't something that was supposed to be generated by Suricata. I'll take a look at the libmagic issue.

With other distros the .mgc file is provided by the magic packages from the distro. Suricata just loads that from the default location (NULL passed to magic_open). Perhaps on OpenWRT an extra step is required, or perhaps another package related to magic should be installed?

Victor Julien wrote:

With other distros the .mgc file is provided by the magic packages from the distro. Suricata just loads that from the default location (NULL passed to magic_open). Perhaps on OpenWRT an extra step is required, or perhaps another package related to magic should be installed?

I'm certain it's a OpenWrt issue and unrelated to the reported bug. I was just doing the due diligence in making sure. As to why OpenWrt isn't generating the mgc file I need to look further into. I see it has provisions for it; I just need to figure out why it isn't doing it. At least I can get it to compile and attempt to load. Once I can manage to see that it actually works, I'll then see if I can find someone who can assist making a rust-lang and cargo package so it can be submitted for inclusion. I'm not sure how many people would use it, given the size it expands the image (roughly 23MB, which puts it outside the size range for SOHO type devices) but I certainly will, and it'll give people a choice.

The limitation reported in the Bug report is with autoconf not being able to do the AC_CHECK_FILE on the host while cross compiling. Everything I've seen replaces the AC_CHECK_FILE/AC_CHECK_FILES calls with other ways to check if CROSS_COMPILE is set.. (reference example: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/commit/81e5b262da643414507a76c4db185d28e81d9a32)

I don't know if this is something Suricata itself can put in the configure.ac or not, but would eliminate the cross compile errors without limiting options like suricata-update (or rust in the future).

I certainly am appreciative on the support I found here in this.

Donald Hoskins wrote:

... I'll then see if I can find someone who can assist making a rust-lang and cargo package so it can be submitted for inclusion. I'm not sure how many people would use it, given the size it expands the image (roughly 23MB, which puts it outside the size range for SOHO type devices) but I certainly will, and it'll give people a choice.

I think rustc/cargo would only be a build time dependency, so it should not lead to large space use on the devices. I assume the cross compile needs to work for our Rust side too. No idea if that will work.

Donald Hoskins wrote:

The limitation reported in the Bug report is with autoconf not being able to do the AC_CHECK_FILE on the host while cross compiling. Everything I've seen replaces the AC_CHECK_FILE/AC_CHECK_FILES calls with other ways to check if CROSS_COMPILE is set.. (reference example: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/commit/81e5b262da643414507a76c4db185d28e81d9a32)

I don't know if this is something Suricata itself can put in the configure.ac or not, but would eliminate the cross compile errors without limiting options like suricata-update (or rust in the future).

I would like us to fix that in the configure.ac, yes.

Victor Julien wrote:

Donald Hoskins wrote:

The limitation reported in the Bug report is with autoconf not being able to do the AC_CHECK_FILE on the host while cross compiling. Everything I've seen replaces the AC_CHECK_FILE/AC_CHECK_FILES calls with other ways to check if CROSS_COMPILE is set.. (reference example: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/commit/81e5b262da643414507a76c4db185d28e81d9a32)

I don't know if this is something Suricata itself can put in the configure.ac or not, but would eliminate the cross compile errors without limiting options like suricata-update (or rust in the future).

I would like us to fix that in the configure.ac, yes.

I figured out the magic.mgc file issue. OpenWrt doesn't install the /usr/share/misc/magic file unless the file package is also installed. Then, the magic.mgc file can be compiled with file -C -m magic.

So, Suricata 4.1.4 is working.

Now, as to the upcoming rust and cargo requirements. Are you suggesting rust and cargo would only be a HOST requirement, and not a TARGET requirement? As in, I'd need rust and cargo installed on my Ubuntu build box, but not necessary in the OpenWrt? So it won't need rust to run once compiled, but will be needed in order to BE compiled?
Am I understand that correctly, or misunderstanding?

If that is the case, it completely invalidates any reservations I have about the 5.0 build.

Great news on the magic side.

Wrt Rust, I think it should only be required on the build host, not on the target.

In Debian its in Build-Depends it seems: https://sources.debian.org/src/suricata/1:4.1.4-2/debian/control/#L16

Victor Julien wrote:

In Debian its in Build-Depends it seems: https://sources.debian.org/src/suricata/1:4.1.4-2/debian/control/#L16

Here is how I bypassed the cross_compiling check in configure.ac. Of course, it is a manual change at the moment.

Suricata Updater check

    if test "$enable_suricata_update" = "yes"; then
      if test "$cross_compiling" != "yes"; then
         AC_CHECK_FILE([$srcdir/suricata-update/setup.py], [
          have_suricata_update="yes"], [])
      else
         AS_IF([test -a "$srcdir/suricata-update/setup.py"], have_suricata_update="yes", [])
      fi
    fi

Rust check

    if test "$cross_compiling" != "yes"; then
      AC_CHECK_FILES([$srcdir/rust/vendor], [have_rust_vendor="yes"])
    else
      AS_IF([test -d "$srcdir/rust/vendor"], have_rust_vendor="yes", [])
    fi

As an OT aside, so I don't continue to clutter up this bug report with, admittedly, Suricata shop talk, that isn't related, if either one of you would like to help me troubleshoot/work through issues with Suricata and Rust in OpenWrt outside of this, please let me know. Email or Google Hangouts would work (grommish@gmail.com)

Victor, I think the fact I'm building this for another device makes the terminology get confusing. The dependency for Debian is there, but anyone using those deps would be building FOR the debian HOST, no? With what I'm trying to do, the TARGET isn't the HOST. While outlining the above changes, I was looking at the rust checking code, and it seems to only be checking the HOST. As long as rustc/cargo aren't using AFTER the compile, then I don't see a reason 5.x couldn't work (except for a rustc might not support mips64). With the above changes to configure.ac, I was able to get it mostly built (I ran into compile issues with libsuricata.a), but I saw where cargo was pulling down files for the build.

FYI, Debian has a bug report that might be related: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=923174

There is a patch attached that will make it into an upcoming upload (https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=923174;filename=cross.patch;msg=5). Maybe this is relevant for this issue?