Task #2757
openimprove protocol detection
Description
Currently protocol detection is a mix of port independent pattern based matching and port depended 'probing parsers'. This has served reasonably well, but also has serious limitations. Protocols that are similar in structure (e.g. smtp, ftp and irc) are not well supported.
The goal of the improvement is to fix these limitations.
Files
Updated by Victor Julien almost 6 years ago
- Related to Feature #273: IRC protocol detection support added
Updated by Victor Julien almost 6 years ago
- Related to Feature #511: Port indepedent protocol identification (nDPI) added
Updated by Victor Julien almost 6 years ago
- Related to Feature #1125: smtp: improve protocol detection added
Updated by Victor Julien almost 6 years ago
- Related to Task #2685: SuriCon 2018 brainstorm added
Updated by Victor Julien over 5 years ago
- Related to Bug #2393: One way TLS traffic not properly identified added
Updated by Victor Julien over 5 years ago
- Status changed from New to Assigned
- Assignee set to Victor Julien
Updated by Victor Julien over 5 years ago
- Related to Bug #2978: IRC traffic parsed by FTP added
Updated by Victor Julien over 5 years ago
- Assignee changed from Victor Julien to Philippe Antoine
Updated by Philippe Antoine over 5 years ago
Do I understand correctly the challenges ?
We want to support detection
- unidirectional (for both client and server sides)
- midstream
- port independent
- similar FTP-IRC-SMTP
- performant
What about evasion techniques ?
Such as my evil FTP client connects to a FTP server, but then first sends a EHLO request (rejected by the server) and then speaks regular FTP
Updated by Victor Julien over 5 years ago
Correct. It should be robust against evasions too. One way we do this (for TCP) is to check if client and server protos match. If not, an event is set.
While port independent is the goal, it might not always be achievable. I could imagine that in some ambiguous cases, like SMTP/FTP, we would ultimately fall back to ports or at least assign some weight to the ports.
Updated by Philippe Antoine over 5 years ago
- There is clearly the server and client protos to match (and not stop on first detection as is done now calling TCPProtoDetect)
- Support when multiple protocols have patterns matching (such as USER for FTP and IRC)
- Check completeness for the current protocols in terms of patterns, probes, both client and server side
- Add a configurable behavior to run all probing parsers on unrecognized trafic whatever the port (and then add lots of protocols in terms of recognition only)
Updated by Philippe Antoine about 5 years ago
There is clearly the server and client protos to match (and not stop on first detection as is done now calling TCPProtoDetect)
In facts, it seems to be done already for TCP (whatever the comment says)
But it seems not done over UDP
Updated by Philippe Antoine about 5 years ago
- Related to deleted (Bug #2393: One way TLS traffic not properly identified)
Updated by Philippe Antoine about 5 years ago
About completeness, here are the protocols referenced in app-layer-protos.h but not implemented :
- IMAP
- MSN
- JABBER
- IRC
Should we implement at least detection for them ?
And here are the other partially (ie TO_CLIENT) missing protocol detections (and current pull requests) :
SMTP : https://github.com/OISF/suricata/pull/3832
DNS : https://github.com/OISF/suricata/pull/4058
MODBUS : https://github.com/OISF/suricata/pull/4082
DNP3 : https://github.com/OISF/suricata/pull/4082
TFTP : https://github.com/OISF/suricata/pull/4082
For all the other protocols and directions, we have have at least one mechanism (whose relevance I did not check here) :
HTTP,TOSERVER,PM
HTTP,TOCLIENT,PM
FTP,TOSERVER,PM
FTP,TOCLIENT,PM
FTPDATA,FLOW,EXPECTATION
SMTP,TOSERVER,PM
TLS,TOSERVER,PM+PP
TLS,TOCLIENT,PM
SSH,TOSERVER,PM
SSH,TOCLIENT,PM
SMB,TOSERVER,PM+PP
SMB,TOCLIENT,PM+PP
DCERPC,TOSERVER,PM+PP+hack?
DCERPC,TOCLIENT,PM+hack?
DNS,TOSERVER,PP
MODBUS,TOSERVER,PP
ENIP,TOSERVER,PP
ENIP,TOCLIENT,PP
DNP3,TOSERVER,PP
NFS,TOSERVER,PP
NFS,TOCLIENT,PP
TFTP,TOSERVER,PP
DHCP,TOSERVER,PP
DHCP,TOCLIENT,PP
IKEV2,TOSERVER,PP
IKEV2,TOCLIENT,PP
KRB5,TOSERVER,PP
KRB5,TOCLIENT,PP
NTP,TOSERVER,PP
NTP,TOCLIENT,PP
Updated by Philippe Antoine about 5 years ago
About TCP in the case of client and server protocol mismatch, the one taken into consideration is the first one to have sent data...
Here is a problematic case :
https://github.com/OISF/suricata-verify/pull/117
I think that, in the case of a protocol mismatch between client and server, we should parse further to confirm which is the protocol...
Any ideas on this ?
Updated by Philippe Antoine about 5 years ago
To sum up a bit my point of view, we need :
- completeness : what is wished ? (ie should we add IRC, MSN and such ?)
- match client and server protocols : for UDP https://github.com/OISF/suricata/pull/4058
- match client and server protocols : do not always trust the server in case of mismatch cf https://github.com/OISF/suricata-verify/pull/117
- multiple protocol recognition : discussion in https://github.com/OISF/suricata/pull/4152
Updated by Philippe Antoine almost 5 years ago
- Related to Feature #2713: protocol detection w/o protocol parsing added
Updated by Victor Julien almost 5 years ago
- Related to Task #3392: Tracking: protocol detection evasions added
Updated by Philippe Antoine over 4 years ago
- Status changed from Assigned to In Review
Updated by Philippe Antoine over 4 years ago
- Related to deleted (Feature #1125: smtp: improve protocol detection)
Updated by Philippe Antoine over 4 years ago
- Blocks Feature #1125: smtp: improve protocol detection added
Updated by Philippe Antoine over 4 years ago
- Target version changed from TBD to 6.0.0beta1
Updated by Philippe Antoine about 4 years ago
- File sshsmb.pcap sshsmb.pcap added
Here is a sample pcap with both SSH and SMB patterns
Updated by Victor Julien about 4 years ago
- Target version changed from 6.0.0beta1 to 6.0.0rc1
Updated by Philippe Antoine about 4 years ago
- Target version changed from 6.0.0rc1 to 7.0.0-beta1
Updated by Philippe Antoine over 3 years ago
Merged https://github.com/OISF/suricata/pull/5773 : debug validation that we cannot have multiple patterns matching at once
Updated by Philippe Antoine over 2 years ago
Now that https://github.com/OISF/suricata/pull/7320 got merged, this ticket requires more discussion about what needs to be done
- Code cleaning ?
- match client and server protocols : do not always trust the server in case of mismatch (retry multiple times) ?
- completeness (detect more protocols) ?
Updated by Philippe Antoine over 2 years ago
- Blocks deleted (Feature #1125: smtp: improve protocol detection)
Updated by Philippe Antoine over 2 years ago
- Related to Feature #1125: smtp: improve protocol detection added
Updated by Philippe Antoine about 2 years ago
- Target version changed from 7.0.0-beta1 to TBD
Updated by Philippe Antoine about 1 year ago
- Related to Feature #6366: pop3 protocol detection added
Updated by Philippe Antoine about 1 year ago
- Related to Bug #2886: IMAP protocol detection is incomplete added
Updated by Victor Julien 10 months ago
- Related to Bug #6591: protodetect: ftp parsed as smtp added