Project

General

Profile

Bug #3073

AC_CHECK_FILE on cross compile

Added by Donald Hoskins 4 months ago. Updated 3 months ago.

Status:
New
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

Generated configure.ac file doesn't check for cross compile flag before initiating a AC_CHECK_FILE

[[https://gcc.gnu.org/bugzilla/show_bug.cgi?id=26046#c3]]

Errors out with:

checking for ./suricata-update/setup.py... configure: error: cannot check for file existence when cross compiling

configure.ac - Line 1467-1470

    if test "$enable_suricata_update" = "yes"; then
      AC_CHECK_FILE([$srcdir/suricata-update/setup.py], [
          have_suricata_update="yes"], [])
    fi

configure.ac - 2312 - 2315

            AC_CHECK_FILES([$srcdir/rust/vendor], [have_rust_vendor="yes"])
            if test "x$have_rust_vendor" = "xyes"; then
                  rust_vendor_comment="" 
            fi


Files

suricata.log (28.6 KB) suricata.log Donald Hoskins, 07/03/2019 07:38 PM
suricata.yaml (65.6 KB) suricata.yaml Donald Hoskins, 07/08/2019 09:06 PM
suricata-build-info.txt (3.53 KB) suricata-build-info.txt Donald Hoskins, 07/09/2019 08:57 PM

History

#1

Updated by Victor Julien 4 months ago

Can you share your steps on how you are crosscompiling? I'd like to see if it can be added to CI.

#2

Updated by Donald Hoskins 4 months ago

Victor Julien wrote:

Can you share your steps on how you are crosscompiling? I'd like to see if it can be added to CI.

I am building it as an OpenWrt package. Full compile fails, so I then attempted as a stand-alone. I've attached the output log of

make -j1 package/network/suricata/compile V=sc 2>&1 | tee suricata.log
for you to check.

Configuring with the following build options:

        --prefix="/usr" \
        --sysconfdir="/etc" \
        --localstatedir=/var \
        --disable-gccmarch-native \
        --enable-lua \
        --enable-geopip \
        --enable-nfqueue \
        --enable-gccprotect \
        --enable-debug \
        --enable-pie \
        --host=mips64
#3

Updated by Andreas Herz 3 months ago

  • Assignee set to OISF Dev
  • Target version set to TBD

A bit OT, but should this result in an official OpenWRT package?

#4

Updated by Donald Hoskins 3 months ago

Andreas Herz wrote:

A bit OT, but should this result in an official OpenWRT package?

As far as I can tell, Suricata isn't a OpenWrt package. I found a starting point for making the package and fixed it to work (aside from the file check on cross compile). My build machine is a x64 Ubuntu, but the target is mips64.

If I can get it to work, I'll package it as it's own git and figure out how to submit/PR it. They support Snort, of course, but I need options in my builds (also an officially unsupported device)

#5

Updated by Donald Hoskins 3 months ago

Andreas Herz wrote:

A bit OT, but should this result in an official OpenWRT package?

I got Suricata to compile and build into OpenWrt. I had to use --disable-suricata-update and --disable-rust because of the CrossCompile file checks outlined above. Compiled, installed, but reports:

root@OpenWrt:/etc/suricata# suricata -c /etc/suricata/suricata.yaml -s /etc/suricata/rules/signatures.rules -i eth0 -vv
8/7/2019 -- 07:12:03 - <Notice> - This is Suricata version 4.1.4 RELEASE
8/7/2019 -- 07:12:03 - <Info> - CPUs/cores online: 2
8/7/2019 -- 07:12:03 - <Error> - [ERRCODE: SC_ERR_MAGIC_LOAD(197)] - magic_load failed: could not find any valid magic files!

This is outside the scope of the bug report, but I figured I'd let you all know progress was made and how I got around it.

I do have libmagic installed on the device.

root@OpenWrt:/usr/lib# ls *mag
lrwxrwxrwx    1 root     root            17 Jan  1  1970 libmagic.so.1 -> libmagic.so.1.0.0
-rwxr-xr-x    1 root     root        145424 Jul  2 22:32 libmagic.so.1.0.0

OpenWrt looks interested in making Suricata a package, but the upcoming changes to 5.x in the Rust/Cargo requirements is going to cause issues. If I can, I'll see about trying to make an accompanying rust-lang and cargo package for it, but I'd like to attempt that only after I get 4.1.4 working as intended.

#6

Updated by Andreas Herz 3 months ago

Can you add the /etc/suricata/suricata.yaml I guess that's just a config setting issue.

And yes the rust part might be more difficult but since we will make it mandatory it would be recommended.

#7

Updated by Donald Hoskins 3 months ago

Andreas Herz wrote:

Can you add the /etc/suricata/suricata.yaml I guess that's just a config setting issue.

And yes the rust part might be more difficult but since we will make it mandatory it would be recommended.

I was just using the default suricata.yaml for testing, but I did attach it. I don't HAVE a magic.mgc file (or any .mgc file) on the device. I'm wondering if it's a libmagic issue, or I'm missing something else. The only configuration options I disabled were listed above, so..

#8

Updated by Andreas Herz 3 months ago

Can you also paste the suricata --build-info output?

It's a bit odd that you didn't get any warnings related to that if there is no libmagic installed. But if you don't need it you can run --disable-libmagic at the configure step.

#9

Updated by Donald Hoskins 3 months ago

Andreas Herz wrote:

Can you also paste the suricata --build-info output?

It's a bit odd that you didn't get any warnings related to that if there is no libmagic installed. But if you don't need it you can run --disable-libmagic at the configure step.

Well, as I said above, I appear to have libmagic installed. It builds, and is present in /usr/lib, but it didn't seem to generate any .mgc file.

Attaching suricata --build-info output

#10

Updated by Andreas Herz 3 months ago

I see, but that's a non suricata issue then and needs to be fixed on OpenWRT side.

#11

Updated by Donald Hoskins 3 months ago

Donald Hoskins wrote:

Andreas Herz wrote:

Can you also paste the suricata --build-info output?

It's a bit odd that you didn't get any warnings related to that if there is no libmagic installed. But if you don't need it you can run --disable-libmagic at the configure step.

Well, as I said above, I appear to have libmagic installed. It builds, and is present in /usr/lib, but it didn't seem to generate any .mgc file.

Attaching suricata --build-info output

Side note: I'm sure it won't effect anything, but when looking at the --build-info, I noticed I hadn't re-added --enable-pie, so I ran off another compile with it. So, it's got PIE support now..

The full configure arguments are below:

CONFIGURE_ARGS = \
        --prefix="/usr" \
        --sysconfdir="/etc" \
        --localstatedir=/var \
        --disable-gccmarch-native \
        --enable-lua \
        --enable-geopip \
        --enable-nfqueue \
        --enable-gccprotect \
        --enable-debug \
        --enable-pie \
        --disable-suricata-update \
        --disable-rust \
        --host=$(ARCH)

#12

Updated by Donald Hoskins 3 months ago

Andreas Herz wrote:

I see, but that's a non suricata issue then and needs to be fixed on OpenWRT side.

Yes. I wanted to ensure that it wasn't something that was supposed to be generated by Suricata. I'll take a look at the libmagic issue.

#13

Updated by Victor Julien 3 months ago

With other distros the .mgc file is provided by the magic packages from the distro. Suricata just loads that from the default location (NULL passed to magic_open). Perhaps on OpenWRT an extra step is required, or perhaps another package related to magic should be installed?

#14

Updated by Donald Hoskins 3 months ago

Victor Julien wrote:

With other distros the .mgc file is provided by the magic packages from the distro. Suricata just loads that from the default location (NULL passed to magic_open). Perhaps on OpenWRT an extra step is required, or perhaps another package related to magic should be installed?

I'm certain it's a OpenWrt issue and unrelated to the reported bug. I was just doing the due diligence in making sure. As to why OpenWrt isn't generating the mgc file I need to look further into. I see it has provisions for it; I just need to figure out why it isn't doing it. At least I can get it to compile and attempt to load. Once I can manage to see that it actually works, I'll then see if I can find someone who can assist making a rust-lang and cargo package so it can be submitted for inclusion. I'm not sure how many people would use it, given the size it expands the image (roughly 23MB, which puts it outside the size range for SOHO type devices) but I certainly will, and it'll give people a choice.

The limitation reported in the Bug report is with autoconf not being able to do the AC_CHECK_FILE on the host while cross compiling. Everything I've seen replaces the AC_CHECK_FILE/AC_CHECK_FILES calls with other ways to check if CROSS_COMPILE is set.. (reference example: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/commit/81e5b262da643414507a76c4db185d28e81d9a32)

I don't know if this is something Suricata itself can put in the configure.ac or not, but would eliminate the cross compile errors without limiting options like suricata-update (or rust in the future).

I certainly am appreciative on the support I found here in this.

#15

Updated by Victor Julien 3 months ago

Donald Hoskins wrote:

... I'll then see if I can find someone who can assist making a rust-lang and cargo package so it can be submitted for inclusion. I'm not sure how many people would use it, given the size it expands the image (roughly 23MB, which puts it outside the size range for SOHO type devices) but I certainly will, and it'll give people a choice.

I think rustc/cargo would only be a build time dependency, so it should not lead to large space use on the devices. I assume the cross compile needs to work for our Rust side too. No idea if that will work.

#16

Updated by Victor Julien 3 months ago

Donald Hoskins wrote:

The limitation reported in the Bug report is with autoconf not being able to do the AC_CHECK_FILE on the host while cross compiling. Everything I've seen replaces the AC_CHECK_FILE/AC_CHECK_FILES calls with other ways to check if CROSS_COMPILE is set.. (reference example: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/commit/81e5b262da643414507a76c4db185d28e81d9a32)

I don't know if this is something Suricata itself can put in the configure.ac or not, but would eliminate the cross compile errors without limiting options like suricata-update (or rust in the future).

I would like us to fix that in the configure.ac, yes.

#17

Updated by Donald Hoskins 3 months ago

Victor Julien wrote:

Donald Hoskins wrote:

The limitation reported in the Bug report is with autoconf not being able to do the AC_CHECK_FILE on the host while cross compiling. Everything I've seen replaces the AC_CHECK_FILE/AC_CHECK_FILES calls with other ways to check if CROSS_COMPILE is set.. (reference example: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/commit/81e5b262da643414507a76c4db185d28e81d9a32)

I don't know if this is something Suricata itself can put in the configure.ac or not, but would eliminate the cross compile errors without limiting options like suricata-update (or rust in the future).

I would like us to fix that in the configure.ac, yes.

I figured out the magic.mgc file issue. OpenWrt doesn't install the /usr/share/misc/magic file unless the file package is also installed. Then, the magic.mgc file can be compiled with file -C -m magic.

So, Suricata 4.1.4 is working.

Now, as to the upcoming rust and cargo requirements. Are you suggesting rust and cargo would only be a HOST requirement, and not a TARGET requirement? As in, I'd need rust and cargo installed on my Ubuntu build box, but not necessary in the OpenWrt? So it won't need rust to run once compiled, but will be needed in order to BE compiled?
Am I understand that correctly, or misunderstanding?

If that is the case, it completely invalidates any reservations I have about the 5.0 build.

#18

Updated by Victor Julien 3 months ago

Great news on the magic side.

Wrt Rust, I think it should only be required on the build host, not on the target.

#20

Updated by Donald Hoskins 3 months ago

Victor Julien wrote:

In Debian its in Build-Depends it seems: https://sources.debian.org/src/suricata/1:4.1.4-2/debian/control/#L16

Here is how I bypassed the cross_compiling check in configure.ac. Of course, it is a manual change at the moment.

Suricata Updater check

    if test "$enable_suricata_update" = "yes"; then
      if test "$cross_compiling" != "yes"; then
         AC_CHECK_FILE([$srcdir/suricata-update/setup.py], [
          have_suricata_update="yes"], [])
      else
         AS_IF([test -a "$srcdir/suricata-update/setup.py"], have_suricata_update="yes", [])
      fi
    fi

Rust check

    if test "$cross_compiling" != "yes"; then
      AC_CHECK_FILES([$srcdir/rust/vendor], [have_rust_vendor="yes"])
    else
      AS_IF([test -d "$srcdir/rust/vendor"], have_rust_vendor="yes", [])
    fi

As an OT aside, so I don't continue to clutter up this bug report with, admittedly, Suricata shop talk, that isn't related, if either one of you would like to help me troubleshoot/work through issues with Suricata and Rust in OpenWrt outside of this, please let me know. Email or Google Hangouts would work ()

Victor, I think the fact I'm building this for another device makes the terminology get confusing. The dependency for Debian is there, but anyone using those deps would be building FOR the debian HOST, no? With what I'm trying to do, the TARGET isn't the HOST. While outlining the above changes, I was looking at the rust checking code, and it seems to only be checking the HOST. As long as rustc/cargo aren't using AFTER the compile, then I don't see a reason 5.x couldn't work (except for a rustc might not support mips64). With the above changes to configure.ac, I was able to get it mostly built (I ran into compile issues with libsuricata.a), but I saw where cargo was pulling down files for the build.

#21

Updated by Sascha Steinbiss 3 months ago

FYI, Debian has a bug report that might be related: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=923174

There is a patch attached that will make it into an upcoming upload (https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=923174;filename=cross.patch;msg=5). Maybe this is relevant for this issue?

Also available in: Atom PDF