Suricata

Probably this is due to missing CAP_IPC_LOCK.

Thanks for the pointer Alex!

Good news.

I stumbled on this article after researching a bit about IPC_MEM_LOCK … shouldn’t affect us since we’re on a newer kernel.

I checked the documentation of mmap and it says the following
https://www.systutorials.com/docs/linux/man/2-mlock/

“ ===
Limits and permissions
In Linux 2.6.8 and earlier, a process must be privileged (CAP_IPC_LOCK) in order to lock memory and the RLIMIT_MEMLOCK soft resource limit defines a limit on how much memory the process may
lock.
####BELOW IS IMPORTANT PART
Since Linux 2.6.9, no limits are placed on the amount of memory that a privileged process can lock and the RLIMIT_MEMLOCK soft resource limit instead defines a limit on how much memory an unprivileged process may ===
Then I looked at the limits configuration

https://software.intel.com/en-us/blogs/2014/12/16/best-known-methods-for-setting-locked-memory-size

By default the limit for memlock is 64Kb… no good.

I changed the ulimit for suricata user to unlimited and it fixed the issue. Now I just need to figure out what the appropriate limit should be but looks like that was the issue.

Thank you for the help and the pointer.

In your case - what kernel version did you try this on - that is working ?

Joel Samaroo wrote:

Thanks for the pointer Alex!

Good news.

I stumbled on this article after researching a bit about CAP_IPC_LOCK … shouldn’t affect us since we’re on a newer kernel.

I checked the documentation of mmap and it says the following
https://www.systutorials.com/docs/linux/man/2-mlock/

“ ===
Limits and permissions
In Linux 2.6.8 and earlier, a process must be privileged (CAP_IPC_LOCK) in order to lock memory and the RLIMIT_MEMLOCK soft resource limit defines a limit on how much memory the process may
lock.
####BELOW IS IMPORTANT PART
Since Linux 2.6.9, no limits are placed on the amount of memory that a privileged process can lock and the RLIMIT_MEMLOCK soft resource limit instead defines a limit on how much memory an unprivileged process may ===
Then I looked at the limits configuration

https://software.intel.com/en-us/blogs/2014/12/16/best-known-methods-for-setting-locked-memory-size

By default the limit for memlock is 64Kb… no good.

I changed the ulimit for suricata user to unlimited and it fixed the issue. Now I just need to figure out what the appropriate limit should be but looks like that was the issue.

Thank you for the help and the pointer.

Hi Peter,

Kernel version 3.10.0

Ok, thanks.
It did not do the trick for me on 4.9.x (changing it to unlimited). Do you mind sharing exactly what you did that it works for you ?

Yes after updating limits.conf the effect will not take place until you either
1) log out and log back in as new user
2) ssh into the host again

In /etc/security/limits.conf
Add the following limit
suricata hard memlock unlimited

You can confirm if it’s been set by running the following to verify

su -s /bin/bash -c ‘ulimit -H -a’ suricata | grep -i lock

That should let you know if the settings have been reloaded.

Note if you do not ssh back in the setting will not likely change. When testing on a box configured as run-level 5 (UI with X Windows) you must log out then log back in as user. This is my understanding of how to trigger limits being updated.

Correction
updating limits.conf the effect will not take place until you either
1) log out and log back in as a user
2) ssh into the host again

I did that - I have actually rebooted.

You may need to set it in the systemd service file? https://github.com/elastic/elasticsearch/issues/22734

Ahhh, yes I also added that line to my systemd suricata.service file. Looked like Peter was starting it from command line, so I didn’t think to mention it!

When i try from command line i'm also not able to start suricata as unprivileged user, but via systemd (with LimitMEMLOCK=infinity) it works.
I checked the following stackexchange article
https://unix.stackexchange.com/questions/345595/how-to-set-ulimits-on-service-with-systemd

and it looks like LimitMEMLOCK maps to ulimit -l (memlock) so it's confusing why that would not work for command line.

1. ulimit -H -l unlimited
2. ulimit -S -l unlimited
3. if [ -e /var/run/suricata.pid ];then rm /var/run/suricata.pid;fi ;/sbin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet

5/4/2019 -- 16:41:36 - <Notice> - all 25 packet processing threads, 6 management threads initialized, engine started.

so if you're manually starting the command and using run-user (not via systemd) then you need to make sure the ulimit for root is also high evenough for memlock.

Please note -- setting the ulimit the way I did it above makes sure it's not persistent -- since i did not define it in any file. To make it persistent make the change in your file.
Moral of the store i suppose it that Systemd seems to override root's ulimit (not necessarily the user's ulimits here). It's an interesting lesson to learn, but it makes sense since all of this is set up before the privs are dropped down.
The other thing I'm wondering now, is why is it that if you are not dropping privs you are not given that error. The limit constraints are still there, so that is still a for mystery for me.

I think it makes sense. When Suricata starts it runs as root. Only after opening the AF_PACKET stuff the privs are dropped. So the limits that apply should be those for the root user.

Is this something we should document or where we can change something on suricata side?

I think it makes sense to document it as an OS Specific requirement (as noted above Ubuntu didn't seem to have any issues) when using run-user: with mmap_locked: together.
Not sure where exactly this should be though. We did document internally in our organization, however I think that other's would benefit from noting the change in some sort of other documentation.