https://redmine.openinfosecfoundation.org/https://redmine.openinfosecfoundation.org/favicon.ico?17011170022019-04-04T11:36:32ZOpen Information Security FoundationSuricata - Bug #2918: Unable to mmap, error Resource temporarily unavailable - err seems OS specifichttps://redmine.openinfosecfoundation.org/issues/2918?journal_id=117262019-04-04T11:36:32ZAlexander Gozman
<ul></ul><p>Probably this is due to missing CAP_IPC_LOCK.</p> Suricata - Bug #2918: Unable to mmap, error Resource temporarily unavailable - err seems OS specifichttps://redmine.openinfosecfoundation.org/issues/2918?journal_id=117272019-04-04T19:07:30ZJoel Samaroo
<ul></ul><p>Thanks for the pointer Alex!</p>
<p>Good news.</p>
<p>I stumbled on this article after researching a bit about IPC_MEM_LOCK … shouldn’t affect us since we’re on a newer kernel.</p>
<p>I checked the documentation of mmap and it says the following<br /><a class="external" href="https://www.systutorials.com/docs/linux/man/2-mlock/">https://www.systutorials.com/docs/linux/man/2-mlock/</a></p>
<p>“
===<br />Limits and permissions<br />In Linux 2.6.8 and earlier, a process must be privileged (CAP_IPC_LOCK) in order to lock memory and the RLIMIT_MEMLOCK soft resource limit defines a limit on how much memory the process may <br />lock.<br />####BELOW IS IMPORTANT PART<br />Since Linux 2.6.9, no limits are placed on the amount of memory that a privileged process can lock and the RLIMIT_MEMLOCK soft resource limit instead defines a limit on how much memory an unprivileged process may
===<br />Then I looked at the limits configuration</p>
<p><a class="external" href="https://software.intel.com/en-us/blogs/2014/12/16/best-known-methods-for-setting-locked-memory-size">https://software.intel.com/en-us/blogs/2014/12/16/best-known-methods-for-setting-locked-memory-size</a></p>
<p>By default the limit for memlock is 64Kb… no good.</p>
<p>I changed the ulimit for suricata user to unlimited and it fixed the issue. Now I just need to figure out what the appropriate limit should be but looks like that was the issue.</p>
<p>Thank you for the help and the pointer.</p> Suricata - Bug #2918: Unable to mmap, error Resource temporarily unavailable - err seems OS specifichttps://redmine.openinfosecfoundation.org/issues/2918?journal_id=117282019-04-04T19:53:56ZPeter Manevpetermanev@gmail.com
<ul></ul><p>In your case - what kernel version did you try this on - that is working ?</p> Suricata - Bug #2918: Unable to mmap, error Resource temporarily unavailable - err seems OS specifichttps://redmine.openinfosecfoundation.org/issues/2918?journal_id=117292019-04-04T19:54:32ZJoel Samaroo
<ul></ul><p>Joel Samaroo wrote:</p>
<blockquote>
<p>Thanks for the pointer Alex!</p>
<p>Good news.</p>
<p>I stumbled on this article after researching a bit about CAP_IPC_LOCK … shouldn’t affect us since we’re on a newer kernel.</p>
<p>I checked the documentation of mmap and it says the following<br /><a class="external" href="https://www.systutorials.com/docs/linux/man/2-mlock/">https://www.systutorials.com/docs/linux/man/2-mlock/</a></p>
<p>“
===<br />Limits and permissions<br />In Linux 2.6.8 and earlier, a process must be privileged (CAP_IPC_LOCK) in order to lock memory and the RLIMIT_MEMLOCK soft resource limit defines a limit on how much memory the process may <br />lock.<br />####BELOW IS IMPORTANT PART<br />Since Linux 2.6.9, no limits are placed on the amount of memory that a privileged process can lock and the RLIMIT_MEMLOCK soft resource limit instead defines a limit on how much memory an unprivileged process may
===<br />Then I looked at the limits configuration</p>
<p><a class="external" href="https://software.intel.com/en-us/blogs/2014/12/16/best-known-methods-for-setting-locked-memory-size">https://software.intel.com/en-us/blogs/2014/12/16/best-known-methods-for-setting-locked-memory-size</a></p>
<p>By default the limit for memlock is 64Kb… no good.</p>
<p>I changed the ulimit for suricata user to unlimited and it fixed the issue. Now I just need to figure out what the appropriate limit should be but looks like that was the issue.</p>
<p>Thank you for the help and the pointer.</p>
</blockquote> Suricata - Bug #2918: Unable to mmap, error Resource temporarily unavailable - err seems OS specifichttps://redmine.openinfosecfoundation.org/issues/2918?journal_id=117302019-04-04T19:57:54ZJoel Samaroo
<ul></ul><p>Hi Peter,</p>
<p>Kernel version 3.10.0</p> Suricata - Bug #2918: Unable to mmap, error Resource temporarily unavailable - err seems OS specifichttps://redmine.openinfosecfoundation.org/issues/2918?journal_id=117352019-04-05T06:19:50ZPeter Manevpetermanev@gmail.com
<ul></ul><p>Ok, thanks.<br />It did not do the trick for me on 4.9.x (changing it to unlimited). Do you mind sharing exactly what you did that it works for you ?</p> Suricata - Bug #2918: Unable to mmap, error Resource temporarily unavailable - err seems OS specifichttps://redmine.openinfosecfoundation.org/issues/2918?journal_id=117362019-04-05T06:46:45ZJoel Samaroo
<ul></ul><p>Yes after updating limits.conf the effect will not take place until you either <br />1) log out and log back in as new user<br />2) ssh into the host again</p>
<p>In /etc/security/limits.conf<br />Add the following limit<br />suricata hard memlock unlimited</p>
<p>You can confirm if it’s been set by running the following to verify</p>
<p>su -s /bin/bash -c ‘ulimit -H -a’ suricata | grep -i lock</p>
<p>That should let you know if the settings have been reloaded.</p>
<p>Note if you do not ssh back in the setting will not likely change. When testing on a box configured as run-level 5 (UI with X Windows) you must log out then log back in as user. This is my understanding of how to trigger limits being updated.</p> Suricata - Bug #2918: Unable to mmap, error Resource temporarily unavailable - err seems OS specifichttps://redmine.openinfosecfoundation.org/issues/2918?journal_id=117372019-04-05T06:51:50ZJoel Samaroo
<ul></ul><p>Correction<br /> updating limits.conf the effect will not take place until you either<br />1) log out and log back in as a user<br />2) ssh into the host again</p> Suricata - Bug #2918: Unable to mmap, error Resource temporarily unavailable - err seems OS specifichttps://redmine.openinfosecfoundation.org/issues/2918?journal_id=117382019-04-05T07:59:37ZPeter Manevpetermanev@gmail.com
<ul></ul><p>I did that - I have actually rebooted.</p> Suricata - Bug #2918: Unable to mmap, error Resource temporarily unavailable - err seems OS specifichttps://redmine.openinfosecfoundation.org/issues/2918?journal_id=117392019-04-05T08:04:18ZVictor Julienvictor@inliniac.net
<ul></ul><p>You may need to set it in the systemd service file? <a class="external" href="https://github.com/elastic/elasticsearch/issues/22734">https://github.com/elastic/elasticsearch/issues/22734</a></p> Suricata - Bug #2918: Unable to mmap, error Resource temporarily unavailable - err seems OS specifichttps://redmine.openinfosecfoundation.org/issues/2918?journal_id=117442019-04-05T13:44:01ZJoel Samaroo
<ul></ul><p>Ahhh, yes I also added that line to my systemd suricata.service file. Looked like Peter was starting it from command line, so I didn’t think to mention it!</p> Suricata - Bug #2918: Unable to mmap, error Resource temporarily unavailable - err seems OS specifichttps://redmine.openinfosecfoundation.org/issues/2918?journal_id=117452019-04-05T14:56:07ZJoel Samaroo
<ul></ul><p>When i try from command line i'm also not able to start suricata as unprivileged user, but via systemd (with LimitMEMLOCK=infinity) it works. <br />I checked the following stackexchange article<br /><a class="external" href="https://unix.stackexchange.com/questions/345595/how-to-set-ulimits-on-service-with-systemd">https://unix.stackexchange.com/questions/345595/how-to-set-ulimits-on-service-with-systemd</a></p>
<p>and it looks like LimitMEMLOCK maps to ulimit -l (memlock) so it's confusing why that would not work for command line.</p> Suricata - Bug #2918: Unable to mmap, error Resource temporarily unavailable - err seems OS specifichttps://redmine.openinfosecfoundation.org/issues/2918?journal_id=117492019-04-05T16:32:39ZJoel Samaroo
<ul><li><strong>File</strong> <a href="/attachments/1647">limits comparison.jpg</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/1647/limits%20comparison.jpg">limits comparison.jpg</a> added</li></ul><p>According to attachment there's definitely some issue that I'm having in my tests that's causing my normal user to cap out at 65536kb of locked mem limit.</p>
<p>Would be interested in what that looks like on your end Peter.</p>
<p>The way i'm getting this info is the following:<br />cat /proc/$(ps -u suricata | tail -n1 | cut -f1 -d\ )/limits</p> Suricata - Bug #2918: Unable to mmap, error Resource temporarily unavailable - err seems OS specifichttps://redmine.openinfosecfoundation.org/issues/2918?journal_id=117512019-04-05T16:43:25ZJoel Samaroo
<ul></ul>So this is interesting<br />if you set root ulimit for memory locking to unlimited (by defualt its 64kb or 65536 bytes) it works fine!
<ol>
<li>ulimit -H -l unlimited</li>
<li>ulimit -S -l unlimited</li>
<li>if [ -e /var/run/suricata.pid ];then rm /var/run/suricata.pid;fi ;/sbin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet</li>
</ol>
<p>5/4/2019 -- 16:41:36 - <Notice> - all 25 packet processing threads, 6 management threads initialized, engine started.</p>
<p>so if you're manually starting the command and using run-user (not via systemd) then you need to make sure the ulimit for root is also high evenough for memlock.</p> Suricata - Bug #2918: Unable to mmap, error Resource temporarily unavailable - err seems OS specifichttps://redmine.openinfosecfoundation.org/issues/2918?journal_id=117522019-04-05T17:06:57ZJoel Samaroo
<ul></ul><p>Please note -- setting the ulimit the way I did it above makes sure it's not persistent -- since i did not define it in any file. To make it persistent make the change in your file. <br />Moral of the store i suppose it that Systemd seems to override root's ulimit (not necessarily the user's ulimits here). It's an interesting lesson to learn, but it makes sense since all of this is set up before the privs are dropped down. <br />The other thing I'm wondering now, is why is it that if you are not dropping privs you are not given that error. The limit constraints are still there, so that is still a for mystery for me.</p> Suricata - Bug #2918: Unable to mmap, error Resource temporarily unavailable - err seems OS specifichttps://redmine.openinfosecfoundation.org/issues/2918?journal_id=117532019-04-05T18:05:37ZVictor Julienvictor@inliniac.net
<ul></ul><p>I think it makes sense. When Suricata starts it runs as root. Only after opening the AF_PACKET stuff the privs are dropped. So the limits that apply should be those for the root user.</p> Suricata - Bug #2918: Unable to mmap, error Resource temporarily unavailable - err seems OS specifichttps://redmine.openinfosecfoundation.org/issues/2918?journal_id=121862019-05-23T22:02:30ZAndreas Herzoisf@herzandreas.de
<ul><li><strong>Assignee</strong> set to <i>Community Ticket</i></li><li><strong>Target version</strong> set to <i>TBD</i></li></ul> Suricata - Bug #2918: Unable to mmap, error Resource temporarily unavailable - err seems OS specifichttps://redmine.openinfosecfoundation.org/issues/2918?journal_id=131202019-07-27T21:19:56ZAndreas Herzoisf@herzandreas.de
<ul></ul><p>Is this something we should document or where we can change something on suricata side?</p> Suricata - Bug #2918: Unable to mmap, error Resource temporarily unavailable - err seems OS specifichttps://redmine.openinfosecfoundation.org/issues/2918?journal_id=147342019-11-22T17:25:39ZJoel Samaroo
<ul></ul><p>I think it makes sense to document it as an OS Specific requirement (as noted above Ubuntu didn't seem to have any issues) when using run-user: with mmap_locked: together.<br />Not sure where exactly this should be though. We did document internally in our organization, however I think that other's would benefit from noting the change in some sort of other documentation.</p>