app-layer-protocol:failed; doesn't match traffic with ALPROTO_UNKNOWN
I'm working on writing some signatures for some malware that uses a custom protocol. In my signatures, I'd like to be able to write "app-layer-protocol:failed;" in order to filter out traffic that has a known protocol associated with it. However, with that logic incorporated to my signatures, some of the signatures fail to fire. Upon inspection of these rule matches with and without that logic, I found that the rules that consistently fire are being "matched" with a protocol as ALPROTO_FAILED, and the ones that are not firing are not matched at all, and thus are labeled as having no app layer protocol, i.e. ALPROTO_UNKNOWN. There is no support for looking for traffic with app-layer-protocol:unknown; either. From my perspective, any traffic that has an "unknown" protocol, would be considered to have failed application layer protocol inspection. Therefore, I would like app-layer-protocol:failed; to match both traffic that has been classified as ALPROTO_UNKNOWN and ALPROTO_FAILED.
A tedious workaround to this thing would be to have something to the effect of
app-layer-protocol:!http; app-layer-protocol:!tls; app-layer-protocol:!smtp; app-layer-protocol:!ftp; ...
You get the idea. Unfortunately, that is a lot of extra logic to include in a signature, and also means the signature is not as easily maintained -- any additional protocols added to suricata would need to be NOT-ed out later.
I'm not sure if this is considered a feature or a bug... I felt it was a bug, as I expected my traffic to match against the signature whether application layer protocol detection failed or... ended up in an unknown state. Those are effectively the same state in my eyes, I understand that those might have different meanings within the code of suricata. Please let me know if I can help by providing any more information, clarifying anything, etc. I don't know how to label this in terms of effort or difficulty either; I ask that you please excuse my ignorance.