Suricata

It looks like we have an issue with protocol detection here. The pattern based protocol detection detects the ftp data body as HTTP, but somehow the expectation based protocol detect later overrides it, but it does not handle the case where the protocol was already set to HTTP, and the state was already initialized as HtpState. Thus later on the Flow::alstate is freed using FTPDataStateFree.

At

#0  AppLayerExpectationHandle (f=0x3501f00, flags=7 '\a') at app-layer-expectation.c:342
#1  0x00000000009a84af in AppLayerProtoDetectPEGetProto (f=0x3501f00, ipproto=6 '\006', flags=7 '\a') at app-layer-detect-proto.c:469
#2  0x00000000009a7242 in AppLayerProtoDetectGetProto (tctx=0x7ffff0500480, f=0x3501f00, buf=0x0, buflen=0, ipproto=6 '\006', flags=7 '\a', reverse_flow=0x7ffff62114f7) at app-layer-detect-proto.c:1581
#3  0x000000000097a60a in TCPProtoDetect (tv=0x3572480, ra_ctx=0x7ffff0500400, app_tctx=0x7ffff0500430, p=0x7ffff04c7840, f=0x3501f00, ssn=0x7ffff05dfc00, stream=0x7ffff6211640, data=0x0, data_len=0, 
    flags=7 '\a') at app-layer.c:330
#4  0x0000000000979fbe in AppLayerHandleTCPData (tv=0x3572480, ra_ctx=0x7ffff0500400, p=0x7ffff04c7840, f=0x3501f00, ssn=0x7ffff05dfc00, stream=0x7ffff6211640, data=0x0, data_len=0, flags=7 '\a')
    at app-layer.c:645
#5  0x0000000000d8d74c in StreamTcpReassembleAppLayer (tv=0x3572480, ra_ctx=0x7ffff0500400, ssn=0x7ffff05dfc00, stream=0x7ffff05dfc98, p=0x7ffff04c7840, dir=UPDATE_DIR_PACKET) at stream-tcp-reassemble.c:1243
#6  0x0000000000d90870 in StreamTcpReassembleHandleSegment (tv=0x3572480, ra_ctx=0x7ffff0500400, ssn=0x7ffff05dfc00, stream=0x7ffff05dfc98, p=0x7ffff04c7840, pq=0x7ffff04e82d0) at stream-tcp-reassemble.c:1923
#7  0x0000000000d4111f in StreamTcpPacket (tv=0x3572480, p=0x7ffff04c7840, stt=0x7ffff05000f0, pq=0x7ffff04e82d0) at stream-tcp.c:4940
#8  0x0000000000d44e86 in StreamTcp (tv=0x3572480, p=0x7ffff04c7840, data=0x7ffff05000f0, pq=0x7ffff04e82d0) at stream-tcp.c:5312
#9  0x0000000000cdf616 in FlowWorkerStreamTCPUpdate (tv=0x3572480, fw=0x7ffff04e82a0, p=0x7ffff04c7840, detect_thread=0x0, timeout=true) at flow-worker.c:370
#10 0x0000000000ce0255 in FlowWorkerFlowTimeout (tv=0x3572480, p=0x7ffff04c7840, fw=0x7ffff04e82a0, detect_thread=0x0) at flow-worker.c:413
#11 0x0000000000ce014c in FlowFinish (tv=0x3572480, f=0x3501f00, fw=0x7ffff04e82a0, detect_thread=0x0) at flow-worker.c:154
#12 0x0000000000cdfde5 in CheckWorkQueue (tv=0x3572480, fw=0x7ffff04e82a0, detect_thread=0x0, counters=0x7ffff6211940, fq=0x7ffff6211960) at flow-worker.c:177
#13 0x0000000000cdfb0f in FlowWorkerProcessInjectedFlows (tv=0x3572480, fw=0x7ffff04e82a0, p=0x3a3b870, detect_thread=0x0) at flow-worker.c:459
#14 0x0000000000cdf22b in FlowWorker (tv=0x3572480, p=0x3a3b870, data=0x7ffff04e82a0) at flow-worker.c:588
#15 0x00000000008ff96f in TmThreadsSlotVarRun (tv=0x3572480, p=0x3a3b870, slot=0x2467130) at tm-threads.c:117
#16 0x0000000000905f1b in TmThreadsSlotProcessPkt (tv=0x3572480, s=0x2467130, p=0x3a3b870) at ./tm-threads.h:195
#17 0x0000000000905ddd in TmThreadTimeoutLoop (tv=0x3572480, s=0x247b5f0) at tm-threads.c:177
#18 0x00000000009056cd in TmThreadsSlotPktAcqLoop (td=0x3572480) at tm-threads.c:330
#19 0x00007ffff7d5c609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#20 0x00007ffff71e4293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

$5 = {src = {address = {address_un_data32 = {3757650186, 0, 0, 0}, address_un_data16 = {12554, 57337, 0, 0, 0, 0, 0, 0}, address_un_data8 = "\n1\371\337", '\000' <repeats 11 times>}}, dst = {address = {
      address_un_data32 = {63907382, 0, 0, 0}, address_un_data16 = {9782, 975, 0, 0, 0, 0, 0, 0}, address_un_data8 = "6&\317\003", '\000' <repeats 11 times>}}, {sp = 49168, icmp_s = {type = 16 '\020', 
      code = 192 '\300'}, esp = {spi = 49168}}, {dp = 51048, icmp_d = {type = 104 'h', code = 199 '\307'}}, proto = 6 '\006', recursion_level = 0 '\000', vlan_id = {0, 0}, use_cnt = 2, vlan_idx = 0 '\000', {{
      ffr_ts = 1 '\001', ffr_tc = 1 '\001'}, ffr = 17 '\021'}, timeout_at = 1624314947, thread_id = {1, 1}, next = 0x0, livedev = 0x0, flow_hash = 1396070381, lastts = {tv_sec = 1624314347, tv_usec = 378567}, 
  timeout_policy = 600, flow_state = 1, tenant_id = 0, probing_parser_toserver_alproto_masks = 0, probing_parser_toclient_alproto_masks = 0, flags = 10076163, file_flags = 1023, protodetect_dp = 0, 
  parent_id = 0, m = {__data = {__lock = 1, __count = 0, __owner = 287858, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, 
    __size = "\001\000\000\000\000\000\000\000rd\004\000\001", '\000' <repeats 26 times>, __align = 1}, protoctx = 0x7ffff05dfc00, protomap = 0 '\000', flow_end_flags = 80 'P', alproto = 1, alproto_ts = 17, 
  alproto_tc = 17, alproto_orig = 0, alproto_expect = 0, de_ctx_version = 0, min_ttl_toserver = 128 '\200', max_ttl_toserver = 128 '\200', min_ttl_toclient = 64 '@', max_ttl_toclient = 64 '@', 
  alparser = 0x7ffff0780480, alstate = 0x7ffff07800f0, sgh_toclient = 0x0, sgh_toserver = 0x0, flowvar = 0x0, fb = 0x0, startts = {tv_sec = 1624314327, tv_usec = 277618}, todstpktcnt = 3, tosrcpktcnt = 3, 
  todstbytecnt = 186, tosrcbytecnt = 642}
(gdb) p *(HtpState *)f->alstate
$6 = {connp = 0x7ffff0781de0, conn = 0x7ffff0781f80, f = 0x3501f00, transaction_cnt = 1, store_tx_id = 0, files_ts = 0x0, files_tc = 0x7ffff0780370, cfg = 0x2400cf8 <cfglist>, flags = 0, events = 1, 
  htp_messages_offset = 1, file_track_id = 1, file_range = 0x0, last_request_data_stamp = 0, last_response_data_stamp = 468}

Note: alproto = 1 which is HTTP

Let's fix the segfault first :
https://github.com/OISF/suricata/pull/6651

Thinking a bit more, I wonder if having a separate expectation list makes sense at all. Instead of an expectation we can also just set up a flow that already has the correct protocol set up. That will be a bigger project though.

Sounds good.
Would we log this flow even if we do not see real packets ?
Could it lead to some evasion ? Like I will pretend waiting for ftp-data on port 80 and then do regular HTTP ?

Yeah lets create a ticket for that.