Suricata

Problem¶

When a Suricata file-based output fails to (re)open its log file, the
logger loses the file and keeps dropping records with no error, no
counter and no recovery - until a restart or SIGHUP.

Failure cases include a full disk (ENOSPC), a temporarily unwritable
filesystem, or the target path becoming unusable at rotation time
(eg. logrotate moves the file and the new one cannot be created).
The current behaviour has several gaps:

• A failed (re)open closes the file and leaves the context dead:
  there is no per-file failure state and no retry, so once the open
  fails the output is gone until the next HUP or a restart - even
  when the underlying problem (eg. disk space) clears seconds later.
• A failed rotation is worse: the previous descriptor is closed
  before the new file is opened. When the new open fails, the old
  file - still open and writable - has already been closed, so every
  record from that point is lost.
• The loss is silent: writes to a closed file are skipped without any
  counter or error message, and the close report only carries
  "output errors" - there is no notion of records dropped because
  the file was unopenable. Write failures on an open file are logged
  once and never repeated, no matter how long the disk stays full.
• The flush path is not guarded against the NULL descriptor left
  after a failed (re)open.

Impact¶

A full disk is a common and transient condition on IDS hosts logging
to eve.json. Today the records are silently dropped, the operator is
not told that logging is down, and recovery requires a restart even
after the disk has space again.

Expected behaviour¶

A failed (re)open should be tracked per file: on a failed rotation
the old descriptor is kept open so writes continue on the old inode;
a rate-limited reopen (1s cadence, driven from both the write path
and the 1s maintenance tick) recovers the file once the path is
openable again; failure logging is backoff-limited so a persistent
ENOSPC does not flood the log; and records dropped while the file is
unopenable are counted and reported at close, separately from
socket drops.