Suricata

FTP tracks two globals with similar names: ftp_config_memcap, the configured
limit that FTPCheckMemcap enforces, and ftp_memcap, a counter of how often
that limit was hit. Two problems follow from allocations and commands
consulting the wrong one of these, or neither. Two more follow from what the
parser and the detect layer do once the memcap denies an allocation, or once a
response line is rejected.

The unix socket commands operate on the hit counter, not the limit.
src/runmode-unix-socket.c registers FTPSetMemcap and FTPMemcapGlobalCounter
as the set/get pair for "ftp", and both of those touch the hit counter. So
"memcap-set ftp <value>" reports success while changing nothing that is
enforced -- there is no way to adjust the FTP memcap at runtime -- and as a
side effect it overwrites the ftp.memcap statistic with the requested byte
size. "memcap-show ftp" and "memcap-list" have the matching problem in
reverse, printing the hit counter formatted as a size:

$ suricata --unix-socket --set app-layer.protocols.ftp.memcap=10mb
$ suricatasc -c "memcap-show ftp" 
{"message": {"value": "unlimited"}, "return": "OK"}

A process that has not yet hit its memcap reports the FTP limit as
"unlimited" no matter what the configuration says. Setting 0 for unlimited
is also rejected, unlike the other memcaps.

Two allocations are charged to ftp.memuse without consulting the memcap.
The transfer command allocated by SCFTPTransferCmdNew, and the response
line payload whose total_size is added to memuse when the wrapper is built,
are both added to ftp.memuse with no memcap check. Running a session whose
server response is ~3000 bytes under app-layer.protocols.ftp.memcap=2048
stores the response and pushes memuse past the configured limit; the next
allocation that does check is then denied. The excess is bounded, but the
memcap is not a ceiling.

ftp.command_data reads past a null pointer once a request copy is denied.
DetectFTPCommandDataGetData decides whether a transaction carries command
data with (tx->request_length - b_len - 1) > 0. request_length is a uint32_t
and b_len a uint8_t, so the subtraction is unsigned: a transaction whose
request allocation was denied has request_length 0, the expression evaluates
to 0xfffffffb, and the buffer reaches the matcher as tx->request + b_len + 1,
an offset from a NULL pointer, with a length of 4294967291. FTPParseRequest
fills in tx->command_descriptor before calling CopyCommandLine, so the denied
copy leaves a valid command code behind and the transaction is inspected like
any other. With a rule using ftp.command_data loaded, a session that pipelines
commands until ftp.memuse reaches a 1024 byte memcap ends the process on
SIGSEGV.

The response truncation state survives a line the parser rejects.
FTPParseResponse clears state->current_line_truncated_tc only after a response
wrapper has been allocated and linked onto the transaction. A line that
SCFTPParseResponseLine rejects -- it returns NULL when the line trims to
nothing -- or whose wrapper allocation the memcap denies, leaves the flag set.
FTPGetLineForDirection then takes the next line for the tail of the truncated
one: it discards that line, clears the flag and returns APP_LAYER_ERROR. The
reply that followed is lost and the replies after it are logged one
transaction late. With app-layer.protocols.ftp.max-line-length=64 and a
100 byte whitespace reply to USER, the 230 answering PASS never appears and
the 221 answering QUIT is logged against PASS:

{"command": "PASS", "command_data": "password", "completion_code": ["221"], "reply": ["Goodbye."], "reply_received": "yes"}
{"command": "QUIT", "reply_received": "no"}

Expected: memcap-set/memcap-show operate on the enforced limit and accept 0
for unlimited; no allocation charged to ftp.memuse bypasses the memcap check;
a transaction whose request was denied presents an empty ftp.command_data
buffer; a rejected response line leaves the truncation state clear, so the
reply that follows is parsed and logged against its own transaction.

Affected versions: 8.0.x and 9.0.0-dev have all four.