Project

General

Profile

Actions

Task #2757

open

improve protocol detection

Added by Victor Julien almost 6 years ago. Updated 4 months ago.

Status:
In Review
Priority:
Normal
Target version:
Effort:
Difficulty:
Label:

Description

Currently protocol detection is a mix of port independent pattern based matching and port depended 'probing parsers'. This has served reasonably well, but also has serious limitations. Protocols that are similar in structure (e.g. smtp, ftp and irc) are not well supported.

The goal of the improvement is to fix these limitations.


Files

sshsmb.pcap (1.84 KB) sshsmb.pcap Philippe Antoine, 07/28/2020 09:24 AM

Related issues 10 (6 open4 closed)

Related to Suricata - Feature #273: IRC protocol detection supportNewCommunity TicketActions
Related to Suricata - Feature #511: Port indepedent protocol identification (nDPI)NewCommunity TicketActions
Related to Suricata - Task #2685: SuriCon 2018 brainstormAssignedVictor JulienActions
Related to Suricata - Bug #2978: IRC traffic parsed by FTPIn ProgressPhilippe AntoineActions
Related to Suricata - Feature #2713: protocol detection w/o protocol parsingClosedPierre ChifflierActions
Related to Suricata - Task #3392: Tracking: protocol detection evasionsNewPhilippe AntoineActions
Related to Suricata - Feature #1125: smtp: improve protocol detectionClosedPhilippe AntoineActions
Related to Suricata - Feature #6366: pop3 protocol detectionClosedPhilippe AntoineActions
Related to Suricata - Bug #2886: IMAP protocol detection is incompleteClosedMahmoud MaatuqActions
Related to Suricata - Bug #6591: protodetect: ftp parsed as smtpNewOISF DevActions
Actions #1

Updated by Victor Julien almost 6 years ago

  • Related to Feature #273: IRC protocol detection support added
Actions #2

Updated by Victor Julien almost 6 years ago

  • Related to Feature #511: Port indepedent protocol identification (nDPI) added
Actions #3

Updated by Victor Julien almost 6 years ago

  • Related to Feature #1125: smtp: improve protocol detection added
Actions #4

Updated by Victor Julien almost 6 years ago

  • Related to Task #2685: SuriCon 2018 brainstorm added
Actions #5

Updated by Victor Julien over 5 years ago

  • Related to Bug #2393: One way TLS traffic not properly identified added
Actions #6

Updated by Victor Julien over 5 years ago

  • Status changed from New to Assigned
  • Assignee set to Victor Julien
Actions #7

Updated by Victor Julien over 5 years ago

  • Related to Bug #2978: IRC traffic parsed by FTP added
Actions #8

Updated by Andreas Herz over 5 years ago

  • Target version set to TBD
Actions #9

Updated by Victor Julien over 5 years ago

  • Assignee changed from Victor Julien to Philippe Antoine
Actions #10

Updated by Philippe Antoine over 5 years ago

Do I understand correctly the challenges ?

We want to support detection
- unidirectional (for both client and server sides)
- midstream
- port independent
- similar FTP-IRC-SMTP
- performant

What about evasion techniques ?
Such as my evil FTP client connects to a FTP server, but then first sends a EHLO request (rejected by the server) and then speaks regular FTP

Actions #11

Updated by Victor Julien over 5 years ago

Correct. It should be robust against evasions too. One way we do this (for TCP) is to check if client and server protos match. If not, an event is set.

While port independent is the goal, it might not always be achievable. I could imagine that in some ambiguous cases, like SMTP/FTP, we would ultimately fall back to ports or at least assign some weight to the ports.

Actions #12

Updated by Philippe Antoine over 5 years ago

So, I see multiple things to do :
  • There is clearly the server and client protos to match (and not stop on first detection as is done now calling TCPProtoDetect)
  • Support when multiple protocols have patterns matching (such as USER for FTP and IRC)
  • Check completeness for the current protocols in terms of patterns, probes, both client and server side
  • Add a configurable behavior to run all probing parsers on unrecognized trafic whatever the port (and then add lots of protocols in terms of recognition only)
Actions #13

Updated by Philippe Antoine about 5 years ago

There is clearly the server and client protos to match (and not stop on first detection as is done now calling TCPProtoDetect)

In facts, it seems to be done already for TCP (whatever the comment says)
But it seems not done over UDP

Actions #14

Updated by Philippe Antoine about 5 years ago

  • Related to deleted (Bug #2393: One way TLS traffic not properly identified)
Actions #15

Updated by Philippe Antoine about 5 years ago

About completeness, here are the protocols referenced in app-layer-protos.h but not implemented :
- IMAP
- MSN
- JABBER
- IRC

Should we implement at least detection for them ?

And here are the other partially (ie TO_CLIENT) missing protocol detections (and current pull requests) :
SMTP : https://github.com/OISF/suricata/pull/3832
DNS : https://github.com/OISF/suricata/pull/4058
MODBUS : https://github.com/OISF/suricata/pull/4082
DNP3 : https://github.com/OISF/suricata/pull/4082
TFTP : https://github.com/OISF/suricata/pull/4082

For all the other protocols and directions, we have have at least one mechanism (whose relevance I did not check here) :
HTTP,TOSERVER,PM
HTTP,TOCLIENT,PM
FTP,TOSERVER,PM
FTP,TOCLIENT,PM
FTPDATA,FLOW,EXPECTATION
SMTP,TOSERVER,PM
TLS,TOSERVER,PM+PP
TLS,TOCLIENT,PM
SSH,TOSERVER,PM
SSH,TOCLIENT,PM
SMB,TOSERVER,PM+PP
SMB,TOCLIENT,PM+PP
DCERPC,TOSERVER,PM+PP+hack?
DCERPC,TOCLIENT,PM+hack?
DNS,TOSERVER,PP
MODBUS,TOSERVER,PP
ENIP,TOSERVER,PP
ENIP,TOCLIENT,PP
DNP3,TOSERVER,PP
NFS,TOSERVER,PP
NFS,TOCLIENT,PP
TFTP,TOSERVER,PP
DHCP,TOSERVER,PP
DHCP,TOCLIENT,PP
IKEV2,TOSERVER,PP
IKEV2,TOCLIENT,PP
KRB5,TOSERVER,PP
KRB5,TOCLIENT,PP
NTP,TOSERVER,PP
NTP,TOCLIENT,PP

Actions #16

Updated by Philippe Antoine about 5 years ago

About TCP in the case of client and server protocol mismatch, the one taken into consideration is the first one to have sent data...
Here is a problematic case :
https://github.com/OISF/suricata-verify/pull/117

I think that, in the case of a protocol mismatch between client and server, we should parse further to confirm which is the protocol...
Any ideas on this ?

Actions #17

Updated by Philippe Antoine about 5 years ago

To sum up a bit my point of view, we need :
- completeness : what is wished ? (ie should we add IRC, MSN and such ?)
- match client and server protocols : for UDP https://github.com/OISF/suricata/pull/4058
- match client and server protocols : do not always trust the server in case of mismatch cf https://github.com/OISF/suricata-verify/pull/117
- multiple protocol recognition : discussion in https://github.com/OISF/suricata/pull/4152

Actions #18

Updated by Philippe Antoine almost 5 years ago

  • Related to Feature #2713: protocol detection w/o protocol parsing added
Actions #19

Updated by Victor Julien almost 5 years ago

  • Related to Task #3392: Tracking: protocol detection evasions added
Actions #21

Updated by Philippe Antoine over 4 years ago

  • Related to deleted (Feature #1125: smtp: improve protocol detection)
Actions #22

Updated by Philippe Antoine over 4 years ago

Actions #23

Updated by Philippe Antoine over 4 years ago

  • Target version changed from TBD to 6.0.0beta1
Actions #24

Updated by Philippe Antoine about 4 years ago

Here is a sample pcap with both SSH and SMB patterns

Actions #25

Updated by Victor Julien about 4 years ago

  • Target version changed from 6.0.0beta1 to 6.0.0rc1
Actions #26

Updated by Philippe Antoine about 4 years ago

  • Target version changed from 6.0.0rc1 to 7.0.0-beta1
Actions #27

Updated by Philippe Antoine over 3 years ago

Merged https://github.com/OISF/suricata/pull/5773 : debug validation that we cannot have multiple patterns matching at once

Actions #28

Updated by Philippe Antoine over 2 years ago

Now that https://github.com/OISF/suricata/pull/7320 got merged, this ticket requires more discussion about what needs to be done
- Code cleaning ?
- match client and server protocols : do not always trust the server in case of mismatch (retry multiple times) ?
- completeness (detect more protocols) ?

Actions #29

Updated by Philippe Antoine over 2 years ago

  • Blocks deleted (Feature #1125: smtp: improve protocol detection)
Actions #30

Updated by Philippe Antoine over 2 years ago

  • Related to Feature #1125: smtp: improve protocol detection added
Actions #31

Updated by Philippe Antoine over 2 years ago

1125 is next

Actions #32

Updated by Philippe Antoine about 2 years ago

  • Target version changed from 7.0.0-beta1 to TBD
Actions #33

Updated by Philippe Antoine about 1 year ago

Actions #34

Updated by Philippe Antoine about 1 year ago

  • Related to Bug #2886: IMAP protocol detection is incomplete added
Actions #35

Updated by Victor Julien 10 months ago

  • Related to Bug #6591: protodetect: ftp parsed as smtp added
Actions #36

Updated by Philippe Antoine 4 months ago

  • Tracker changed from Feature to Task
Actions

Also available in: Atom PDF