Support for Unix Socket
It would be great if suricata could log straight to a unix socket without having to use Barnyard2 as the intermediary. That would help integration with our product PacketFence.
#6 Updated by Mike Pomraning over 6 years ago
The attached patch implements unix stream socket logging as an extension to the regular file logging we have today. Those file-based output modules which do not handle their own rotation (
http) can now log either to a socket or, as before, to a plain file.
Configuration looks like this:
outputs: - fast: enabled: yes filename: fast.log # interpreted based on 'type', below type: socket # defaults to 'regular' (old behavior) append: yes # ignored for sockets
The socket output is a single,
fdopen'd socket connection posing as the regular
FILE * the current code expects. *util-logopenfile.[ch]@ consolidates YAML configuration handling and file/socket opening in one place. A new error, SC_ERR_SOCKET, is introduced.
Appears to work for me on my own system, logging to a trivial dummy listener. Break it and let me know.
Un-mutexed output should be possible, as several threads could locklessly share a dgram socket, or each could have its own connection to a stream socket.
#10 Updated by Vivek Rajagopalan over 6 years ago
Here is a patch for supporting unix domain socket alert output in Suricata.
- Compatible with Snort 2.9.x "-A unsock" - If your app can read snort unsock, this will work too
- Removed the mutex and network/host order code - both are not needed for datagram sockets (as Mike mentioned above)
A new config param called unix-socket. If the filename is an absolute path that will be used for the socket, if not the filename is treated relative to default_log_dir.
+ # unix sockets + - unix-socket: + enabled: yes + filename : /tmp/suricata.sock + +
I have tested it on my own system and it works.
I am checking out Mikes patch now.
#11 Updated by Victor Julien over 6 years ago
I have reviewed Vivek's patch. Looking very good overall. Here are some comments:
1. please use strlcpy and strlcat in every place strncpy, strcpy, etc are used.
2. indent looks pretty messed up in the redmine patch viewer, please use 4 spaces indent everywhere.
3. "remote" in UnixSocketAlert looks like it can be initialized in the thread init function and then stored in the thread local memory. Saves some cycles per alert.
4. If sendto fails, it does "return TM_ECODE_FAILED;". This will make Suricata close the thread and try to respawn it. In what conditions can sendto fail?
5. minor style comment: overall in Suricata we do most checks explicit, so instead of if (p->payload_len) we do if (p->payload_len > 0), instead of if (p->icmp4h) we do if (p->icmpv4h != NULL).
Vivek will do first review on Mike's patch, I'll review the end result :)
Thanks Vivek and Mike, appreciate it!
#12 Updated by Vivek Rajagopalan over 6 years ago
Thanks for the comments, Victor,
Here is the new patch based on your comments and also some minor edits from my side.
sendto can fail if listening socket does not exist or does not have write permission for the user running Suricata. I changed the failure case to log an error but return TM_ECODE_OK. Once the permissions are corrected, the error messages stop and everything will be normal again. One, maybe acceptable risk with this approach is a flood of error messages.
stream vs dgram
For unix domain sockets, there appears to be no gain in using stream. dgram is reliable and atomic.
I am reviewing Mikes code now. Looking good.
#13 Updated by Mike Pomraning over 6 years ago
Vivek Rajagopalan wrote:
I will also check out this patch tomorrow. Is your patch compatible with Snort 2.9 unsock output ?
(Vivek, thanks and sorry I didn't see your response.)
Re: stream v. dgram, as long as the typical alert can fit in an AF_LOCAL dgram, dgram is fine by me. (Indeed, I think that paves the way to remove output mutexen, since whole datagram sends should never be interleaved.)
alert_unixsock compatibility, no, my patch is not compatible. FWIW and IIUC, I find having a different output format just because you have a different output device (file, socket, network socket, POSIX message queue, whatever) a little weird and certainly un-UNIXy. For compatibility, if needed, I'd suggest a new output format module with configurable output device.
#14 Updated by Vivek Rajagopalan over 6 years ago
I tested out the patch and it works fine. I also could not find any issues with Suricata coding style based on Victor's feedback to my patch.
A tiny ruby program
require 'socket' raise "Usage : usock sock_path" unless ARGV.size == 1 socket = UNIXServer.new(ARGV.shift).accept # Just read messages from sock and print out while(line = socket.gets) do puts line end
I sent alert_fastlog to the unix socket on which the above script was running and it printed alerts as expected.
Comment to Mike
It may help to ignore SIGPIPE - especially since we are fdopen-ing the stream unix socket. At the moment, Suricata will trip on SIGPIPE and crash if the process listening on the unix socket closes socket. To test this run the above script and press Ctrl+C while alerts are coming in. This is only an issue with STREAM. Alternately you can move to DGRAM but then we need to write a generic replacement for fwrite as well.
#15 Updated by Victor Julien over 6 years ago
Mike Pomraning wrote:
For compatibility, if needed, I'd suggest a new output format module with configurable output device.
I tend to agree here. It would also lead to a lot less code duplication.
The fast log and syslog outputs are pretty much the same, think we can merge those and add the unix socket output while at it?
#16 Updated by Mike Pomraning over 6 years ago
It may help to ignore SIGPIPE [...] This is only an issue with STREAM.
Alternately you can move to DGRAM but then we need to write a generic replacement for fwrite as well.
Yes, let me think about that but I think SIG_IGNoring PIPE on stream sockets may be best in the near term.
On my Linux box, a DGRAM masquerading as a FILE* isn't suitable:
fprintf(fdopen(dgram_sock), ...) cheerfully ignores ENOTCONN if the reader should die -- a situation less desirable IMHO than having suricata itself die. (I assume you meant
Digression: Long term, yes, IMHO an appropriate approach for suricata's file-like output is "polymorphic:" opaque output handles provide function pointers that effectively implement out->printf (if needed), out->write and (when a formatting module gets serious about performance) out->writev as appropriate to the output channel. Then under the hood we can pass MSG_NOSIGNAL if needed, detect/recover from errors, etc., without being constrained by
stdio.h, and the caller won't have a clue whether the output is a file, a socket, shared memory, whatever.
#17 Updated by Mike Pomraning over 5 years ago
Updated patch for AF_UNIX sockets, both dgram and stream. Note that I can't test it right now - my test environment is hosed right now. :)
Backward-compatible YAML config extension:
append: yes # ignored for UNIX sockets
filetype: regular # default, also 'unix_dgram' and 'unix_stream'
#18 Updated by Mike Pomraning over 5 years ago
- to use
- of stream and dgram fast log on production system
Dec 7 17:03:08 qualys suricata:  7/12/2011 -- 17:03:08 - (util-logopenfile.c:162) <Info> (SCConfLogOpenGeneric) -- fast output device (unix_dgram) initialized: foo.sock ... Dec 7 17:11:45 qualys suricata:  7/12/2011 -- 17:11:45 - (util-logopenfile.c:162) <Info> (SCConfLogOpenGeneric) -- fast output device (unix_stream) initialized: foo.sock