Suricata

When considering the future of our lua integration, it might be interesting to keep an eye on https://github.com/dibyendumajumdar/ravi

From Suricon 2020 brainstorm: Python suggested as a scripting language.

Discussion in 2020 brainstorm got stuck on the same issue wrt lua versions. Vendoring in is something distro's won't like, even through the language seems to be meant to be vendored in. We heard that "luajit is dead". Homepage hasn't been updated in ages, so it certainly doesn't look very alive.

One new point that was brought up is that lua has been successfully sandboxed in other tools, so we could at least improve that side of things.

Some more thoughts post-brainstorm...

Sascha posted the link https://github.com/dbohdan/embedded-scripting-languages, which has a lot of options for embedding a scripting language. Clearly we should no go down the path of creating our own with so many options out there. But we should also think about the ecosystem of the language. We could choose something that is just a language, with no libraries, etc. Suricata has to export all functionality into the script runtime, and a library base could be built up from users. This is more or less what Zeek has to do.

With Lua, for better or worse you can have access to the Lua ecosystem (Luarocks, etc). Likewise for Javascript, Python, etc.

Jo did bring up that Lua can be sandboxed and argued that Lua is almost perfect for what we need it for (the most common use of Lua to extend rules), which might be true, but only true if we can lock to specific version, which may mean vendoring. Or ending up in a situation where its not available everywhere if a distribution drops support for the version we lock to.

Jason Ish wrote in #note-10:

Jo did bring up that Lua can be sandboxed and argued that Lua is almost perfect for what we need it for (the most common use of Lua to extend rules), which might be true, but only true if we can lock to specific version, which may mean vendoring. Or ending up in a situation where its not available everywhere if a distribution drops support for the version we lock to.

One issue with vendoring in a version is that if the runtime is no longer supported upstream we will have to support it for security fixes and general bugs as well. I find this a compelling argument for choosing something that is still maintained. This then means you commit to a release cadence of that project. So then I think a framework that cares about backwards compatibility would be strongly preferred.

After more discussion at Suricon 2021 I'm starting to lean towards to sticking with lua, however overhauling it completely:

• vendor a specific version in (probably the highest current stable), so rule writers can target a specific version
• disallow modules by default (or at all). This would mean Suricata needs to expose more functionality
• do proper sandboxing, both for security and performance. This would mean Suricata needs to expose more functionality
• enable it by default so rule vendors can rely on it to be there

We can then bump the vendored major version on Suricata's major version updates. This might break compatibility of existing scripts, but this is not unlike changes we make to other parts of Suricatas input and output.

Suricon crowd seemed to agree with this approach. There was some mention that with more recent Lua versions the perf delta with luajit has gotten smaller and ppl generally seemed to prefer regular lua.