Feature #250

Support for Unix Socket

Added by Francois Gaudreault over 3 years ago. Updated almost 2 years ago.

Status:ClosedStart date:12/03/2010
Priority:LowDue date:
Assignee:OISF Community% Done:

0%

Category:-
Target version:1.2rc1

Description

It would be great if suricata could log straight to a unix socket without having to use Barnyard2 as the intermediary. That would help integration with our product PacketFence.

suricata-1.1beta1-alert-socket.diff Magnifier - UNIX socket support in 1.1beta1 (16.7 KB) Mike Pomraning, 03/13/2011 07:37 AM

unixsocket.patch Magnifier - Patch file for unix socket support (18.5 KB) Vivek Rajagopalan, 03/15/2011 01:26 AM

unixsocket.2.patch Magnifier - Patch incorporating initial comments (18.9 KB) Vivek Rajagopalan, 03/15/2011 06:09 AM

fr250-20111206.patch Magnifier - Updated patch, more functionality. (23 KB) Mike Pomraning, 12/06/2011 03:59 PM

suricata-fr250.diff Magnifier - Updated patch (strlcpy) (23.7 KB) Mike Pomraning, 12/07/2011 02:18 PM

History

#1 Updated by Victor Julien over 3 years ago

I see how it can be useful for some use cases. I don't think we will have the resources to implement this any time soon though. Can you assist us with this?

#2 Updated by Francois Gaudreault over 3 years ago

Sure, I can test any patches on our test system if needed.

#3 Updated by Victor Julien over 3 years ago

  • Priority changed from Normal to Low

That is useful, although those patches won't come from our (paid) devs anytime soon due to the reasons I stated above.

#4 Updated by Francois Gaudreault over 3 years ago

Victor,

That's fine, it's not a life threatening situation :P This is the only feature missing in suricata to be able to support your IDS, and I guess we can use Barnyard2 during that wait time.

Thanks

#5 Updated by Victor Julien over 3 years ago

  • Assignee set to OISF Community

#6 Updated by Mike Pomraning about 3 years ago

Overview
The attached patch implements unix stream socket logging as an extension to the regular file logging we have today. Those file-based output modules which do not handle their own rotation (fast, drop, debug and http) can now log either to a socket or, as before, to a plain file.

Configuration looks like this:

  outputs:
    - fast:
      enabled: yes
      filename: fast.log # interpreted based on 'type', below
      type: socket       # defaults to 'regular' (old behavior)
      append: yes        # ignored for sockets

Implementation
The socket output is a single, fdopen'd socket connection posing as the regular FILE * the current code expects. *util-logopenfile.[ch]@ consolidates YAML configuration handling and file/socket opening in one place. A new error, SC_ERR_SOCKET, is introduced.

Testing
Appears to work for me on my own system, logging to a trivial dummy listener. Break it and let me know.

Future Directions?
Un-mutexed output should be possible, as several threads could locklessly share a dgram socket, or each could have its own connection to a stream socket.

#7 Updated by Francois Gaudreault about 3 years ago

Thanks!

I'll test this patch this week and I'll keep you posted.

#8 Updated by Vivek Rajagopalan about 3 years ago

Hi Mike,

I will also check out this patch tomorrow. Is your patch compatible with Snort 2.9 unsock output ?

#9 Updated by Vivek Rajagopalan about 3 years ago

Hi Mike,

I will also check out this patch tomorrow. Is your patch compatible with Snort 2.9 unsock output ?

#10 Updated by Vivek Rajagopalan about 3 years ago

Here is a patch for supporting unix domain socket alert output in Suricata.

Doc

  • Compatible with Snort 2.9.x "-A unsock" - If your app can read snort unsock, this will work too
  • Removed the mutex and network/host order code - both are not needed for datagram sockets (as Mike mentioned above)

Use

A new config param called unix-socket. If the filename is an absolute path that will be used for the socket, if not the filename is treated relative to default_log_dir.

+  # unix sockets
+  - unix-socket:
+      enabled: yes
+      filename : /tmp/suricata.sock
+
+

I have tested it on my own system and it works.

I am checking out Mikes patch now.

#11 Updated by Victor Julien about 3 years ago

I have reviewed Vivek's patch. Looking very good overall. Here are some comments:

1. please use strlcpy and strlcat in every place strncpy, strcpy, etc are used.
2. indent looks pretty messed up in the redmine patch viewer, please use 4 spaces indent everywhere.
3. "remote" in UnixSocketAlert looks like it can be initialized in the thread init function and then stored in the thread local memory. Saves some cycles per alert.
4. If sendto fails, it does "return TM_ECODE_FAILED;". This will make Suricata close the thread and try to respawn it. In what conditions can sendto fail?
5. minor style comment: overall in Suricata we do most checks explicit, so instead of if (p->payload_len) we do if (p->payload_len > 0), instead of if (p->icmp4h) we do if (p->icmpv4h != NULL).

Vivek will do first review on Mike's patch, I'll review the end result :)

Thanks Vivek and Mike, appreciate it!

#12 Updated by Vivek Rajagopalan about 3 years ago

Thanks for the comments, Victor,

Here is the new patch based on your comments and also some minor edits from my side.

some notes

sendto can fail if listening socket does not exist or does not have write permission for the user running Suricata. I changed the failure case to log an error but return TM_ECODE_OK. Once the permissions are corrected, the error messages stop and everything will be normal again. One, maybe acceptable risk with this approach is a flood of error messages.

stream vs dgram

For unix domain sockets, there appears to be no gain in using stream. dgram is reliable and atomic.

I am reviewing Mikes code now. Looking good.

#13 Updated by Mike Pomraning about 3 years ago

Vivek Rajagopalan wrote:

Hi Mike,

I will also check out this patch tomorrow. Is your patch compatible with Snort 2.9 unsock output ?

(Vivek, thanks and sorry I didn't see your response.)

Re: stream v. dgram, as long as the typical alert can fit in an AF_LOCAL dgram, dgram is fine by me. (Indeed, I think that paves the way to remove output mutexen, since whole datagram sends should never be interleaved.)

Re: unsock/alert_unixsock compatibility, no, my patch is not compatible. FWIW and IIUC, I find having a different output format just because you have a different output device (file, socket, network socket, POSIX message queue, whatever) a little weird and certainly un-UNIXy. For compatibility, if needed, I'd suggest a new output format module with configurable output device.

#14 Updated by Vivek Rajagopalan about 3 years ago

I tested out the patch and it works fine. I also could not find any issues with Suricata coding style based on Victor's feedback to my patch.

Test
A tiny ruby program

require 'socket'

raise "Usage : usock sock_path" unless ARGV.size == 1

socket  = UNIXServer.new(ARGV.shift).accept

# Just read messages from sock and print out 
while(line = socket.gets) do
   puts line
end

I sent alert_fastlog to the unix socket on which the above script was running and it printed alerts as expected.

Comment to Mike

It may help to ignore SIGPIPE - especially since we are fdopen-ing the stream unix socket. At the moment, Suricata will trip on SIGPIPE and crash if the process listening on the unix socket closes socket. To test this run the above script and press Ctrl+C while alerts are coming in. This is only an issue with STREAM. Alternately you can move to DGRAM but then we need to write a generic replacement for fwrite as well.

#15 Updated by Victor Julien about 3 years ago

Mike Pomraning wrote:

For compatibility, if needed, I'd suggest a new output format module with configurable output device.

I tend to agree here. It would also lead to a lot less code duplication.

The fast log and syslog outputs are pretty much the same, think we can merge those and add the unix socket output while at it?

#16 Updated by Mike Pomraning about 3 years ago

It may help to ignore SIGPIPE [...] This is only an issue with STREAM.
Alternately you can move to DGRAM but then we need to write a generic replacement for fwrite as well.

Yes, let me think about that but I think SIG_IGNoring PIPE on stream sockets may be best in the near term.

On my Linux box, a DGRAM masquerading as a FILE* isn't suitable: fprintf(fdopen(dgram_sock), ...) cheerfully ignores ENOTCONN if the reader should die -- a situation less desirable IMHO than having suricata itself die. (I assume you meant fprintf, not fwrite.)

Digression: Long term, yes, IMHO an appropriate approach for suricata's file-like output is "polymorphic:" opaque output handles provide function pointers that effectively implement out->printf (if needed), out->write and (when a formatting module gets serious about performance) out->writev as appropriate to the output channel. Then under the hood we can pass MSG_NOSIGNAL if needed, detect/recover from errors, etc., without being constrained by stdio.h, and the caller won't have a clue whether the output is a file, a socket, shared memory, whatever.

#17 Updated by Mike Pomraning over 2 years ago

Updated patch for AF_UNIX sockets, both dgram and stream. Note that I can't test it right now - my test environment is hosed right now. :)

Backward-compatible YAML config extension:

- foo
filename: foo.log
append: yes # ignored for UNIX sockets
filetype: regular # default, also 'unix_dgram' and 'unix_stream'

#18 Updated by Mike Pomraning over 2 years ago

  1. Updated patch to use strlcpy
  2. Tested functionality of stream and dgram fast log on production system
Dec  7 17:03:08 qualys suricata: [15692] 7/12/2011 -- 17:03:08 - (util-logopenfile.c:162) <Info> (SCConfLogOpenGeneric) -- fast output device (unix_dgram) initialized: foo.sock
...
Dec  7 17:11:45 qualys suricata: [15762] 7/12/2011 -- 17:11:45 - (util-logopenfile.c:162) <Info> (SCConfLogOpenGeneric) -- fast output device (unix_stream) initialized: foo.sock

(alerts omitted)

#19 Updated by Victor Julien over 2 years ago

Mike's patches have been applied to the git master, thanks a lot Mike!

This adds unixsock output to all the line based outputs, but not to unified2.

#20 Updated by Victor Julien almost 2 years ago

  • Status changed from New to Closed
  • Target version set to 1.2rc1

Closing this as it's been merged.

Also available in: Atom PDF