Project

General

Profile

Actions

Task #3307

open

Research: evaluate future of lua support in Suricata

Added by Victor Julien almost 2 years ago. Updated 10 months ago.

Status:
New
Priority:
Normal
Assignee:
Target version:
Effort:
Difficulty:
Label:

Description

Suricata has had support for Lua for a long time. Originally we only supported luajit, but later this was extended to regular lua. Luajit was giving us lots of issues and some design limitations. Due to luajit targeting lua 5.1, we've worked with that for the regular lua as well. Lua 5.1 is starting to be considered legacy, and recent versions of distributions are not including it anymore.

One problem that we never really considered is that lua versions include breaking changes. So it is tricky to support all versions of lua in a single script (or maybe impossible). This means that a generic way to ship lua rules, for example as part of a ruleset, is also going to be difficult or even impossible. Unless we are willing to drop support for luajit, we can't move away from the lua 5.1 'standard'.

It looks like one of the main use cases for lua is gaming engines and it seems that these generally vendor lua in, so that they have a single version to work with. We could consider something like this. It would bring overhead and I could imagine distro's not being excited about this.

A drawback of lua is the fragile ecosystem in which many modules exist, but quality of them varies wildly. Also various methods for installing those modules exist. For new users this is quite confusing. Despite these drawbacks the modules have made some really interesting script possible, either for real time inspection or more focused at offline pcap analysis.

At Suricon 2019 it was suggested that perhaps a more modern technology like WASM would be a better fit for us.


Related issues

Related to Task #3288: Suricon 2019 brainstormNewVictor JulienActions
Related to Task #3329: Research: WASM as a Lua alternative and for dynamically loadable modulesAssignedJason IshActions
Related to Task #4097: Suricon 2020 brainstormNewVictor JulienActions
Related to Feature #1971: lua: make mandatoryNewOISF DevActions
Actions #1

Updated by Victor Julien almost 2 years ago

  • Parent task deleted (#3288)
Actions #2

Updated by Victor Julien almost 2 years ago

  • Related to Task #3288: Suricon 2019 brainstorm added
Actions #3

Updated by Victor Julien almost 2 years ago

  • Assignee set to OISF Dev
  • Target version set to TBD
Actions #4

Updated by Victor Julien almost 2 years ago

  • Related to Task #3329: Research: WASM as a Lua alternative and for dynamically loadable modules added
Actions #5

Updated by Victor Julien almost 2 years ago

When considering the future of our lua integration, it might be interesting to keep an eye on https://github.com/dibyendumajumdar/ravi

Actions #6

Updated by Victor Julien 11 months ago

  • Related to Task #4097: Suricon 2020 brainstorm added
Actions #7

Updated by Victor Julien 10 months ago

Actions #8

Updated by Jeff Lucovsky 10 months ago

From Suricon 2020 brainstorm: Python suggested as a scripting language.

Actions #9

Updated by Victor Julien 10 months ago

Discussion in 2020 brainstorm got stuck on the same issue wrt lua versions. Vendoring in is something distro's won't like, even through the language seems to be meant to be vendored in. We heard that "luajit is dead". Homepage hasn't been updated in ages, so it certainly doesn't look very alive.

One new point that was brought up is that lua has been successfully sandboxed in other tools, so we could at least improve that side of things.

Actions #10

Updated by Jason Ish 10 months ago

Some more thoughts post-brainstorm...

Sascha posted the link https://github.com/dbohdan/embedded-scripting-languages, which has a lot of options for embedding a scripting language. Clearly we should no go down the path of creating our own with so many options out there. But we should also think about the ecosystem of the language. We could choose something that is just a language, with no libraries, etc. Suricata has to export all functionality into the script runtime, and a library base could be built up from users. This is more or less what Zeek has to do.

With Lua, for better or worse you can have access to the Lua ecosystem (Luarocks, etc). Likewise for Javascript, Python, etc.

Jo did bring up that Lua can be sandboxed and argued that Lua is almost perfect for what we need it for (the most common use of Lua to extend rules), which might be true, but only true if we can lock to specific version, which may mean vendoring. Or ending up in a situation where its not available everywhere if a distribution drops support for the version we lock to.

Actions #11

Updated by Victor Julien 10 months ago

Jason Ish wrote in #note-10:

Jo did bring up that Lua can be sandboxed and argued that Lua is almost perfect for what we need it for (the most common use of Lua to extend rules), which might be true, but only true if we can lock to specific version, which may mean vendoring. Or ending up in a situation where its not available everywhere if a distribution drops support for the version we lock to.

One issue with vendoring in a version is that if the runtime is no longer supported upstream we will have to support it for security fixes and general bugs as well. I find this a compelling argument for choosing something that is still maintained. This then means you commit to a release cadence of that project. So then I think a framework that cares about backwards compatibility would be strongly preferred.

Actions

Also available in: Atom PDF