Project

General

Profile

Feature #2757

improve protocol detection

Added by Victor Julien over 2 years ago. Updated 5 months ago.

Status:
In Review
Priority:
Normal
Target version:
Effort:
Difficulty:
Label:

Description

Currently protocol detection is a mix of port independent pattern based matching and port depended 'probing parsers'. This has served reasonably well, but also has serious limitations. Protocols that are similar in structure (e.g. smtp, ftp and irc) are not well supported.

The goal of the improvement is to fix these limitations.


Files

sshsmb.pcap (1.84 KB) sshsmb.pcap Philippe Antoine, 07/28/2020 09:24 AM

Related issues

Related to Feature #273: IRC protocol detection supportNewCommunity TicketActions
Related to Feature #511: Port indepedent protocol identificationNewCommunity TicketActions
Related to Task #2685: SuriCon 2018 brainstormNewVictor JulienActions
Related to Bug #2978: IRC traffic parsed by FTPNewPhilippe AntoineActions
Related to Feature #2713: protocol detection w/o protocol parsingFeedbackPierre ChifflierActions
Related to Task #3392: Tracking: protocol detection evasionsNewActions
Blocks Feature #1125: smtp: improve protocol detectionFeedbackPhilippe AntoineActions
#1

Updated by Victor Julien over 2 years ago

  • Related to Feature #273: IRC protocol detection support added
#2

Updated by Victor Julien over 2 years ago

  • Related to Feature #511: Port indepedent protocol identification added
#3

Updated by Victor Julien over 2 years ago

  • Related to Feature #1125: smtp: improve protocol detection added
#4

Updated by Victor Julien over 2 years ago

  • Related to Task #2685: SuriCon 2018 brainstorm added
#5

Updated by Victor Julien over 2 years ago

  • Related to Bug #2393: One way TLS traffic not properly identified added
#6

Updated by Victor Julien over 2 years ago

  • Status changed from New to Assigned
  • Assignee set to Victor Julien
#7

Updated by Victor Julien about 2 years ago

  • Related to Bug #2978: IRC traffic parsed by FTP added
#8

Updated by Andreas Herz about 2 years ago

  • Target version set to TBD
#9

Updated by Victor Julien about 2 years ago

  • Assignee changed from Victor Julien to Philippe Antoine
#10

Updated by Philippe Antoine almost 2 years ago

Do I understand correctly the challenges ?

We want to support detection
- unidirectional (for both client and server sides)
- midstream
- port independent
- similar FTP-IRC-SMTP
- performant

What about evasion techniques ?
Such as my evil FTP client connects to a FTP server, but then first sends a EHLO request (rejected by the server) and then speaks regular FTP

#11

Updated by Victor Julien almost 2 years ago

Correct. It should be robust against evasions too. One way we do this (for TCP) is to check if client and server protos match. If not, an event is set.

While port independent is the goal, it might not always be achievable. I could imagine that in some ambiguous cases, like SMTP/FTP, we would ultimately fall back to ports or at least assign some weight to the ports.

#12

Updated by Philippe Antoine almost 2 years ago

So, I see multiple things to do :
  • There is clearly the server and client protos to match (and not stop on first detection as is done now calling TCPProtoDetect)
  • Support when multiple protocols have patterns matching (such as USER for FTP and IRC)
  • Check completeness for the current protocols in terms of patterns, probes, both client and server side
  • Add a configurable behavior to run all probing parsers on unrecognized trafic whatever the port (and then add lots of protocols in terms of recognition only)
#13

Updated by Philippe Antoine almost 2 years ago

There is clearly the server and client protos to match (and not stop on first detection as is done now calling TCPProtoDetect)

In facts, it seems to be done already for TCP (whatever the comment says)
But it seems not done over UDP

#14

Updated by Philippe Antoine almost 2 years ago

  • Related to deleted (Bug #2393: One way TLS traffic not properly identified)
#15

Updated by Philippe Antoine almost 2 years ago

About completeness, here are the protocols referenced in app-layer-protos.h but not implemented :
- IMAP
- MSN
- JABBER
- IRC

Should we implement at least detection for them ?

And here are the other partially (ie TO_CLIENT) missing protocol detections (and current pull requests) :
SMTP : https://github.com/OISF/suricata/pull/3832
DNS : https://github.com/OISF/suricata/pull/4058
MODBUS : https://github.com/OISF/suricata/pull/4082
DNP3 : https://github.com/OISF/suricata/pull/4082
TFTP : https://github.com/OISF/suricata/pull/4082

For all the other protocols and directions, we have have at least one mechanism (whose relevance I did not check here) :
HTTP,TOSERVER,PM
HTTP,TOCLIENT,PM
FTP,TOSERVER,PM
FTP,TOCLIENT,PM
FTPDATA,FLOW,EXPECTATION
SMTP,TOSERVER,PM
TLS,TOSERVER,PM+PP
TLS,TOCLIENT,PM
SSH,TOSERVER,PM
SSH,TOCLIENT,PM
SMB,TOSERVER,PM+PP
SMB,TOCLIENT,PM+PP
DCERPC,TOSERVER,PM+PP+hack?
DCERPC,TOCLIENT,PM+hack?
DNS,TOSERVER,PP
MODBUS,TOSERVER,PP
ENIP,TOSERVER,PP
ENIP,TOCLIENT,PP
DNP3,TOSERVER,PP
NFS,TOSERVER,PP
NFS,TOCLIENT,PP
TFTP,TOSERVER,PP
DHCP,TOSERVER,PP
DHCP,TOCLIENT,PP
IKEV2,TOSERVER,PP
IKEV2,TOCLIENT,PP
KRB5,TOSERVER,PP
KRB5,TOCLIENT,PP
NTP,TOSERVER,PP
NTP,TOCLIENT,PP

#16

Updated by Philippe Antoine almost 2 years ago

About TCP in the case of client and server protocol mismatch, the one taken into consideration is the first one to have sent data...
Here is a problematic case :
https://github.com/OISF/suricata-verify/pull/117

I think that, in the case of a protocol mismatch between client and server, we should parse further to confirm which is the protocol...
Any ideas on this ?

#17

Updated by Philippe Antoine almost 2 years ago

To sum up a bit my point of view, we need :
- completeness : what is wished ? (ie should we add IRC, MSN and such ?)
- match client and server protocols : for UDP https://github.com/OISF/suricata/pull/4058
- match client and server protocols : do not always trust the server in case of mismatch cf https://github.com/OISF/suricata-verify/pull/117
- multiple protocol recognition : discussion in https://github.com/OISF/suricata/pull/4152

#18

Updated by Philippe Antoine over 1 year ago

  • Related to Feature #2713: protocol detection w/o protocol parsing added
#19

Updated by Victor Julien over 1 year ago

  • Related to Task #3392: Tracking: protocol detection evasions added
#21

Updated by Philippe Antoine over 1 year ago

  • Related to deleted (Feature #1125: smtp: improve protocol detection)
#22

Updated by Philippe Antoine over 1 year ago

#23

Updated by Philippe Antoine about 1 year ago

  • Target version changed from TBD to 6.0.0beta1
#24

Updated by Philippe Antoine 11 months ago

Here is a sample pcap with both SSH and SMB patterns

#25

Updated by Victor Julien 11 months ago

  • Target version changed from 6.0.0beta1 to 6.0.0rc1
#26

Updated by Philippe Antoine 10 months ago

  • Target version changed from 6.0.0rc1 to 7.0rc1
#27

Updated by Philippe Antoine 5 months ago

Merged https://github.com/OISF/suricata/pull/5773 : debug validation that we cannot have multiple patterns matching at once

Also available in: Atom PDF