Suricata

Do you have Suricata dropping privileges and running as another user?

In this particular test case, no. It continues running as root, and all commands here are issued as root.

I could not replicate this with git master or Suricata 4.0.0-rc2 built from the archive. The only way I could replicate it was to add --user on the command line to have Suricata drop privileges to a non-root user, or the run-as configuration file section.

When dropping privileges this is intended behaviour as Suricata not longer has enough privileges to remove the PID file it created. It is our understanding that this is the best practice when dealing with PID files.

Can you show me more info like the ps listing of Suricata while running, and the permissions of the socket file?

Could it be a side effect of having another Suricata running (actually sniffing traffic, etc.) at the same time?
To be fair, root owns the job and the socket, and the pid file doesn't disappear with a kill <pid> from root so I'm not sure how this would be related to dropped privileges (we do have another suricata running doing real work, that drops privileges, but not this one.

Permissions on the socket

root@suricata-test:# ls -al /var/run/suricata/suricata-test-command.socket
srw-r----- 1 root suricata 0 Jul 20 18:54 /var/run/suricata/suricata-test-command.socket

ps listing:

root@zombie-lab.cam:/usr/local/google/home/duaneh# ps -ef | grep suricata | grep -v grep
suricata  76609  25951 99 16:46 ?        03:17:34 /usr/bin/suricata -c /etc/suricata/suricata.yaml --af-packet --user suricata --group suricata -F /etc/suricata/bpf.conf
root     118425      1  5 18:54 ?        00:00:10 /usr/bin/suricata -c /etc/suricata/suricata-socket.yaml --unix-socket -D

friendly ping? Any other data I can provide to help?

No, I have not been able to replicate with 4.0.0-rc or 4.0.0. I to have another instance running with af-packet, as user suricata (run from systemd).

Can you try not running daemon mode? See what the exit code is? Maybe its failing on exit before removing the PID file.

I am fairly certain I'm having this issue.

I'm running Suricata 5.0.0 on Ubuntu 16.04 LTS installed from the apt repositories.
We run in af-packet from systemd as root.
I have ~55 identical devices (hardware, Ubuntu version, patch level, config file) standardized with Ansible.

We've only seen this occur at 2 of our locations (odd considering they're identical and update on the same regular schedule). We do signature updates with oinkmaster and after the job runs (cron as root) it kicks off a kill -USR2 $(pidof suricata) it's at this point that it seems to enter the failed state. When we investigate we see the service status as a green active (exited) state. If we restart the service we get an entry in suricata.log: "<Error> - [ERRCODE: SC_ERR_INITIALIZATION(45)] - pid file '/var/run/suricata.pid' exists but appears stale. Make sure Suricata is not running and then remove /var/run/suricata.pid. Aborting!"

Stopping the service, blowing away the stale pid as described in the log, and starting the service again clears the issue.

Let me know what I can do to help further.

Dylan Walter wrote:

I am fairly certain I'm having this issue.

I'm running Suricata 5.0.0 on Ubuntu 16.04 LTS installed from the apt repositories.
We run in af-packet from systemd as root.
I have ~55 identical devices (hardware, Ubuntu version, patch level, config file) standardized with Ansible.

We've only seen this occur at 2 of our locations (odd considering they're identical and update on the same regular schedule). We do signature updates with oinkmaster and after the job runs (cron as root) it kicks off a kill -USR2 $(pidof suricata) it's at this point that it seems to enter the failed state. When we investigate we see the service status as a green active (exited) state. If we restart the service we get an entry in suricata.log: "<Error> - [ERRCODE: SC_ERR_INITIALIZATION(45)] - pid file '/var/run/suricata.pid' exists but appears stale. Make sure Suricata is not running and then remove /var/run/suricata.pid. Aborting!"

Stopping the service, blowing away the stale pid as described in the log, and starting the service again clears the issue.

Let me know what I can do to help further.

Are you using our provided systemd unit file or creating your own? If using your own, can you add it here please?

Dylan Walter wrote:

I'm fairly certain it's the provided one, but I'm attaching my /etc/init.d/suricata anyway.

This looks like an issue with the init script which is not provided by Suricata itself, but instead of this package.

Suricata does contain a sample Systemd unit file that handles this case. I'll try to ping the appropriate people to see where to take this.

New issue created, https://redmine.openinfosecfoundation.org/issues/3330 for this issue as I believe its specific to the init file.