Project

General

Profile

Bug #2918

Unable to mmap, error Resource temporarily unavailable - err seems OS specific

Added by Peter Manev 6 months ago. Updated about 2 months ago.

Status:
New
Priority:
Normal
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

Based on some previous discussion with Victor.

If we run Suricata with some non root user and specifically enable and use the mmap-locked option - Suricata does not start.
If use the same command but without the user (aka not run it as user "--user=mychemicalromance" but simply as "root") there is no err.
I can reproduce this on Stretch/Buster. But it seems the same is not an issue on Ubuntu.

 /usr/bin/suricata -c /etc/suricata/suricata.yaml --af-packet  -v --user=mychemicalromance --set="af-packet.1.mmap-locked = yes" 
[2540] 3/4/2019 -- 23:53:25 - (suricata.c:1058) <Notice> (LogVersion) -- This is Suricata version 5.0.0-dev (rev 69d0d484e)

[2540] 3/4/2019 -- 23:53:39 - (util-conf.c:115) <Info> (ConfUnixSocketIsEnable) -- Running in live mode, activating unix socket
[2540] 3/4/2019 -- 23:53:39 - (unix-manager.c:131) <Info> (UnixNew) -- Using unix socket file '/var/run/suricata/suricata-command.socket'
[2540] 3/4/2019 -- 23:53:39 - (tm-threads.c:2157) <Notice> (TmThreadWaitOnThreadInit) -- all 2 packet processing threads, 4 management threads initialized, engine started.
[2549] 3/4/2019 -- 23:53:39 - (source-af-packet.c:1934) <Error> (AFPSetupRing) -- [ERRCODE: SC_ERR_MEM_ALLOC(1)] - Unable to mmap, error Resource temporarily unavailable
[2549] 3/4/2019 -- 23:53:39 - (source-af-packet.c:1497) <Error> (ReceiveAFPLoop) -- [ERRCODE: SC_ERR_AFP_CREATE(190)] - Couldn't init AF_PACKET socket, fatal error
[2540] 3/4/2019 -- 23:53:39 - (tm-threads.c:2074) <Error> (TmThreadCheckThreadState) -- [ERRCODE: SC_ERR_FATAL(171)] - thread W#01-enp0s3 failed

Files

limits comparison.jpg (128 KB) limits comparison.jpg Joel Samaroo, 04/05/2019 04:29 PM

History

#1

Updated by Alexander Gozman 6 months ago

Probably this is due to missing CAP_IPC_LOCK.

#2

Updated by Joel Samaroo 6 months ago

Thanks for the pointer Alex!

Good news.

I stumbled on this article after researching a bit about IPC_MEM_LOCK … shouldn’t affect us since we’re on a newer kernel.

I checked the documentation of mmap and it says the following
https://www.systutorials.com/docs/linux/man/2-mlock/

“ ===
Limits and permissions
In Linux 2.6.8 and earlier, a process must be privileged (CAP_IPC_LOCK) in order to lock memory and the RLIMIT_MEMLOCK soft resource limit defines a limit on how much memory the process may
lock.
####BELOW IS IMPORTANT PART
Since Linux 2.6.9, no limits are placed on the amount of memory that a privileged process can lock and the RLIMIT_MEMLOCK soft resource limit instead defines a limit on how much memory an unprivileged process may ===
Then I looked at the limits configuration

https://software.intel.com/en-us/blogs/2014/12/16/best-known-methods-for-setting-locked-memory-size

By default the limit for memlock is 64Kb… no good.

I changed the ulimit for suricata user to unlimited and it fixed the issue. Now I just need to figure out what the appropriate limit should be but looks like that was the issue.

Thank you for the help and the pointer.

#3

Updated by Peter Manev 6 months ago

In your case - what kernel version did you try this on - that is working ?

#4

Updated by Joel Samaroo 6 months ago

Joel Samaroo wrote:

Thanks for the pointer Alex!

Good news.

I stumbled on this article after researching a bit about CAP_IPC_LOCK … shouldn’t affect us since we’re on a newer kernel.

I checked the documentation of mmap and it says the following
https://www.systutorials.com/docs/linux/man/2-mlock/

“ ===
Limits and permissions
In Linux 2.6.8 and earlier, a process must be privileged (CAP_IPC_LOCK) in order to lock memory and the RLIMIT_MEMLOCK soft resource limit defines a limit on how much memory the process may
lock.
####BELOW IS IMPORTANT PART
Since Linux 2.6.9, no limits are placed on the amount of memory that a privileged process can lock and the RLIMIT_MEMLOCK soft resource limit instead defines a limit on how much memory an unprivileged process may ===
Then I looked at the limits configuration

https://software.intel.com/en-us/blogs/2014/12/16/best-known-methods-for-setting-locked-memory-size

By default the limit for memlock is 64Kb… no good.

I changed the ulimit for suricata user to unlimited and it fixed the issue. Now I just need to figure out what the appropriate limit should be but looks like that was the issue.

Thank you for the help and the pointer.

#5

Updated by Joel Samaroo 6 months ago

Hi Peter,

Kernel version 3.10.0

#6

Updated by Peter Manev 6 months ago

Ok, thanks.
It did not do the trick for me on 4.9.x (changing it to unlimited). Do you mind sharing exactly what you did that it works for you ?

#7

Updated by Joel Samaroo 6 months ago

Yes after updating limits.conf the effect will not take place until you either
1) log out and log back in as new user
2) ssh into the host again

In /etc/security/limits.conf
Add the following limit
suricata hard memlock unlimited

You can confirm if it’s been set by running the following to verify

su -s /bin/bash -c ‘ulimit -H -a’ suricata | grep -i lock

That should let you know if the settings have been reloaded.

Note if you do not ssh back in the setting will not likely change. When testing on a box configured as run-level 5 (UI with X Windows) you must log out then log back in as user. This is my understanding of how to trigger limits being updated.

#8

Updated by Joel Samaroo 6 months ago

Correction
updating limits.conf the effect will not take place until you either
1) log out and log back in as a user
2) ssh into the host again

#9

Updated by Peter Manev 6 months ago

I did that - I have actually rebooted.

#10

Updated by Victor Julien 6 months ago

You may need to set it in the systemd service file? https://github.com/elastic/elasticsearch/issues/22734

#11

Updated by Joel Samaroo 6 months ago

Ahhh, yes I also added that line to my systemd suricata.service file. Looked like Peter was starting it from command line, so I didn’t think to mention it!

#12

Updated by Joel Samaroo 6 months ago

When i try from command line i'm also not able to start suricata as unprivileged user, but via systemd (with LimitMEMLOCK=infinity) it works.
I checked the following stackexchange article
https://unix.stackexchange.com/questions/345595/how-to-set-ulimits-on-service-with-systemd

and it looks like LimitMEMLOCK maps to ulimit -l (memlock) so it's confusing why that would not work for command line.

#13

Updated by Joel Samaroo 6 months ago

According to attachment there's definitely some issue that I'm having in my tests that's causing my normal user to cap out at 65536kb of locked mem limit.

Would be interested in what that looks like on your end Peter.

The way i'm getting this info is the following:
cat /proc/$(ps -u suricata | tail -n1 | cut -f1 -d\ )/limits

#14

Updated by Joel Samaroo 6 months ago

So this is interesting
if you set root ulimit for memory locking to unlimited (by defualt its 64kb or 65536 bytes) it works fine!
  1. ulimit -H -l unlimited
  2. ulimit -S -l unlimited
  3. if [ -e /var/run/suricata.pid ];then rm /var/run/suricata.pid;fi ;/sbin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet

5/4/2019 -- 16:41:36 - <Notice> - all 25 packet processing threads, 6 management threads initialized, engine started.

so if you're manually starting the command and using run-user (not via systemd) then you need to make sure the ulimit for root is also high evenough for memlock.

#15

Updated by Joel Samaroo 6 months ago

Please note -- setting the ulimit the way I did it above makes sure it's not persistent -- since i did not define it in any file. To make it persistent make the change in your file.
Moral of the store i suppose it that Systemd seems to override root's ulimit (not necessarily the user's ulimits here). It's an interesting lesson to learn, but it makes sense since all of this is set up before the privs are dropped down.
The other thing I'm wondering now, is why is it that if you are not dropping privs you are not given that error. The limit constraints are still there, so that is still a for mystery for me.

#16

Updated by Victor Julien 6 months ago

I think it makes sense. When Suricata starts it runs as root. Only after opening the AF_PACKET stuff the privs are dropped. So the limits that apply should be those for the root user.

#17

Updated by Andreas Herz 4 months ago

  • Assignee set to Community Ticket
  • Target version set to TBD
#18

Updated by Andreas Herz about 2 months ago

Is this something we should document or where we can change something on suricata side?

Also available in: Atom PDF