Bug #2918
openUnable to mmap, error Resource temporarily unavailable - err seems OS specific
Description
Based on some previous discussion with Victor.
If we run Suricata with some non root user and specifically enable and use the mmap-locked option - Suricata does not start.
If use the same command but without the user (aka not run it as user "--user=mychemicalromance" but simply as "root") there is no err.
I can reproduce this on Stretch/Buster. But it seems the same is not an issue on Ubuntu.
/usr/bin/suricata -c /etc/suricata/suricata.yaml --af-packet -v --user=mychemicalromance --set="af-packet.1.mmap-locked = yes" [2540] 3/4/2019 -- 23:53:25 - (suricata.c:1058) <Notice> (LogVersion) -- This is Suricata version 5.0.0-dev (rev 69d0d484e) [2540] 3/4/2019 -- 23:53:39 - (util-conf.c:115) <Info> (ConfUnixSocketIsEnable) -- Running in live mode, activating unix socket [2540] 3/4/2019 -- 23:53:39 - (unix-manager.c:131) <Info> (UnixNew) -- Using unix socket file '/var/run/suricata/suricata-command.socket' [2540] 3/4/2019 -- 23:53:39 - (tm-threads.c:2157) <Notice> (TmThreadWaitOnThreadInit) -- all 2 packet processing threads, 4 management threads initialized, engine started. [2549] 3/4/2019 -- 23:53:39 - (source-af-packet.c:1934) <Error> (AFPSetupRing) -- [ERRCODE: SC_ERR_MEM_ALLOC(1)] - Unable to mmap, error Resource temporarily unavailable [2549] 3/4/2019 -- 23:53:39 - (source-af-packet.c:1497) <Error> (ReceiveAFPLoop) -- [ERRCODE: SC_ERR_AFP_CREATE(190)] - Couldn't init AF_PACKET socket, fatal error [2540] 3/4/2019 -- 23:53:39 - (tm-threads.c:2074) <Error> (TmThreadCheckThreadState) -- [ERRCODE: SC_ERR_FATAL(171)] - thread W#01-enp0s3 failed
Files
Updated by Alexander Gozman over 5 years ago
Probably this is due to missing CAP_IPC_LOCK.
Updated by Joel Samaroo over 5 years ago
Thanks for the pointer Alex!
Good news.
I stumbled on this article after researching a bit about IPC_MEM_LOCK … shouldn’t affect us since we’re on a newer kernel.
I checked the documentation of mmap and it says the following
https://www.systutorials.com/docs/linux/man/2-mlock/
“
===
Limits and permissions
In Linux 2.6.8 and earlier, a process must be privileged (CAP_IPC_LOCK) in order to lock memory and the RLIMIT_MEMLOCK soft resource limit defines a limit on how much memory the process may
lock.
####BELOW IS IMPORTANT PART
Since Linux 2.6.9, no limits are placed on the amount of memory that a privileged process can lock and the RLIMIT_MEMLOCK soft resource limit instead defines a limit on how much memory an unprivileged process may
===
Then I looked at the limits configuration
https://software.intel.com/en-us/blogs/2014/12/16/best-known-methods-for-setting-locked-memory-size
By default the limit for memlock is 64Kb… no good.
I changed the ulimit for suricata user to unlimited and it fixed the issue. Now I just need to figure out what the appropriate limit should be but looks like that was the issue.
Thank you for the help and the pointer.
Updated by Peter Manev over 5 years ago
In your case - what kernel version did you try this on - that is working ?
Updated by Joel Samaroo over 5 years ago
Joel Samaroo wrote:
Thanks for the pointer Alex!
Good news.
I stumbled on this article after researching a bit about CAP_IPC_LOCK … shouldn’t affect us since we’re on a newer kernel.
I checked the documentation of mmap and it says the following
https://www.systutorials.com/docs/linux/man/2-mlock/“ ===
Limits and permissions
In Linux 2.6.8 and earlier, a process must be privileged (CAP_IPC_LOCK) in order to lock memory and the RLIMIT_MEMLOCK soft resource limit defines a limit on how much memory the process may
lock.
####BELOW IS IMPORTANT PART
Since Linux 2.6.9, no limits are placed on the amount of memory that a privileged process can lock and the RLIMIT_MEMLOCK soft resource limit instead defines a limit on how much memory an unprivileged process may ===
Then I looked at the limits configurationhttps://software.intel.com/en-us/blogs/2014/12/16/best-known-methods-for-setting-locked-memory-size
By default the limit for memlock is 64Kb… no good.
I changed the ulimit for suricata user to unlimited and it fixed the issue. Now I just need to figure out what the appropriate limit should be but looks like that was the issue.
Thank you for the help and the pointer.
Updated by Peter Manev over 5 years ago
Ok, thanks.
It did not do the trick for me on 4.9.x (changing it to unlimited). Do you mind sharing exactly what you did that it works for you ?
Updated by Joel Samaroo over 5 years ago
Yes after updating limits.conf the effect will not take place until you either
1) log out and log back in as new user
2) ssh into the host again
In /etc/security/limits.conf
Add the following limit
suricata hard memlock unlimited
You can confirm if it’s been set by running the following to verify
su -s /bin/bash -c ‘ulimit -H -a’ suricata | grep -i lock
That should let you know if the settings have been reloaded.
Note if you do not ssh back in the setting will not likely change. When testing on a box configured as run-level 5 (UI with X Windows) you must log out then log back in as user. This is my understanding of how to trigger limits being updated.
Updated by Joel Samaroo over 5 years ago
Correction
updating limits.conf the effect will not take place until you either
1) log out and log back in as a user
2) ssh into the host again
Updated by Victor Julien over 5 years ago
You may need to set it in the systemd service file? https://github.com/elastic/elasticsearch/issues/22734
Updated by Joel Samaroo over 5 years ago
Ahhh, yes I also added that line to my systemd suricata.service file. Looked like Peter was starting it from command line, so I didn’t think to mention it!
Updated by Joel Samaroo over 5 years ago
When i try from command line i'm also not able to start suricata as unprivileged user, but via systemd (with LimitMEMLOCK=infinity) it works.
I checked the following stackexchange article
https://unix.stackexchange.com/questions/345595/how-to-set-ulimits-on-service-with-systemd
and it looks like LimitMEMLOCK maps to ulimit -l (memlock) so it's confusing why that would not work for command line.
Updated by Joel Samaroo over 5 years ago
- File limits comparison.jpg limits comparison.jpg added
According to attachment there's definitely some issue that I'm having in my tests that's causing my normal user to cap out at 65536kb of locked mem limit.
Would be interested in what that looks like on your end Peter.
The way i'm getting this info is the following:
cat /proc/$(ps -u suricata | tail -n1 | cut -f1 -d\ )/limits
Updated by Joel Samaroo over 5 years ago
if you set root ulimit for memory locking to unlimited (by defualt its 64kb or 65536 bytes) it works fine!
- ulimit -H -l unlimited
- ulimit -S -l unlimited
- if [ -e /var/run/suricata.pid ];then rm /var/run/suricata.pid;fi ;/sbin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet
5/4/2019 -- 16:41:36 - <Notice> - all 25 packet processing threads, 6 management threads initialized, engine started.
so if you're manually starting the command and using run-user (not via systemd) then you need to make sure the ulimit for root is also high evenough for memlock.
Updated by Joel Samaroo over 5 years ago
Please note -- setting the ulimit the way I did it above makes sure it's not persistent -- since i did not define it in any file. To make it persistent make the change in your file.
Moral of the store i suppose it that Systemd seems to override root's ulimit (not necessarily the user's ulimits here). It's an interesting lesson to learn, but it makes sense since all of this is set up before the privs are dropped down.
The other thing I'm wondering now, is why is it that if you are not dropping privs you are not given that error. The limit constraints are still there, so that is still a for mystery for me.
Updated by Victor Julien over 5 years ago
I think it makes sense. When Suricata starts it runs as root. Only after opening the AF_PACKET stuff the privs are dropped. So the limits that apply should be those for the root user.
Updated by Andreas Herz over 5 years ago
- Assignee set to Community Ticket
- Target version set to TBD
Updated by Andreas Herz over 5 years ago
Is this something we should document or where we can change something on suricata side?
Updated by Joel Samaroo almost 5 years ago
I think it makes sense to document it as an OS Specific requirement (as noted above Ubuntu didn't seem to have any issues) when using run-user: with mmap_locked: together.
Not sure where exactly this should be though. We did document internally in our organization, however I think that other's would benefit from noting the change in some sort of other documentation.