Project

General

Profile

Actions

Feature #511

open

Port indepedent protocol identification

Added by David André about 9 years ago. Updated over 2 years ago.

Status:
New
Priority:
Normal
Target version:
Effort:
Difficulty:
Label:

Description

nDPI open-source GPL library allows to detect the protocol no matter what port is used.

It could be used to implement other Protocol keywords for suricata

http://www.ntop.org/products/ndpi/

Protocols supported according to nDPI documentation:
FTP POP SMTP IMAP DNS IPP HTTP MDNS NTP NETBIOS NFS SSDP BGP SNMP XDMCP SMB SYSLOG DHCP PostgreSQL MySQL TDS DirectDownloadLink I23V5 AppleJuice DirectConnect Socrates WinMX MANOLITO PANDO Filetopia iMESH Kontiki OpenFT Kazaa/Fasttrack Gnutella eDonkey Bittorrent (Extended) OFF AVI Flash OGG MPEG QuickTime RealMedia Windowsmedia MMS XBOX QQ MOVE RTSP Feidian Icecast PPLive PPStream Zattoo SHOUTCast SopCast TVAnts TVUplayer VeohTV QQLive Thunder/Webthunder Soulseek GaduGadu IRC Popo Jabber MSN Oscar Yahoo Battlefield Quake Second Life Steam Halflife2 World of Warcraft Telnet STUN IPSEC GRE ICMP IGMP EGP SCTP OSPF IP in IP RTP RDP VNC PCAnywhere SSL SSH USENET MGCP IAX TFTP AFP StealthNet Aimini SIP Truphone ICMPv6 DHCPv6 Armagetron CrossFire Dofus Fiesta Florensia Guildwars HTTP Application Activesync Kerberos LDAP MapleStory msSQL PPTP WARCRAFT3 World of Kung Fu MEEBO FaceBook Twitter DropBox Gmail Google Maps YouTube Skype Google DCE RPC NetFlow_IPFIX sFlow HTTP Connect (SSL over HTTP) HTTP Proxy Netflix Citrix CitrixOnline/GotoMeeting Apple (iMessage, FaceTime…) Webex WhatsApp Apple iCloud Viber Apple iTunes Radius


Related issues

Related to Feature #2757: improve protocol detectionIn ReviewPhilippe AntoineActions
Actions #1

Updated by Victor Julien about 9 years ago

  • Target version set to TBD

OpenDPI/nDPI is licensed LGPLv3, Suricata is GPLv2. According to http://www.gnu.org/licenses/gpl-faq.html#AllCompatibility these are incompatible.

Actions #2

Updated by Victor Julien about 9 years ago

  • Assignee set to Victor Julien
Actions #3

Updated by Victor Julien about 9 years ago

In the process of asking for legal advice on this.

Actions #5

Updated by Andreas Herz almost 6 years ago

Any update on the legal issue? the lib from wand looks not really up to date

Actions #6

Updated by Vito Piserchia over 5 years ago

I'll try to put and add more information about different protocol inspection projects, for now:

  • libprotoident
  • nDPI

Generalities (contact page, main development site, etc etc)

libprotoident

Description:

(from the project page)

Libprotoident is a library that performs application layer protocol identification for flows. Unlike many techniques that require capturing the entire packet payload, only the first four bytes of payload sent in each direction, the size of the first payload-bearing packet in each direction and the TCP or UDP port numbers for the flow are used by libprotoident.

nDPI

Description

(from the project page):

nDPI is a ntop-maintained superset of the popular OpenDPI library. Released under the LGPL license, its goal is to extend the original library by adding new protocols that are otherwise available only on the paid version of OpenDPI. In addition to Unix platforms, we also support Windows, in order to provide you a cross-platform DPI experience. Furthermore, we have modified nDPI do be more suitable for traffic monitoring applications, by disabling specific features that slow down the DPI engine while being them un-necessary for network traffic monitoring.

Specificities

libprotoident

  • Language: C++
  • Protocol Identification:
    • Payload Matches (mostly), pattern matching on the first four bytes of the payload on each direction of the traffic
    • Payload Size
    • Port Number, used in case of ambiguity and only for well known ports
    • IP Matching, very few cases
    • protocols are checked in order depending on the confidence the author have on the rules and the popularity of the protocol
  • Multi thread support: no, but can be added in the user application
  • Test Pcap: none

nDPI

  • Language: C
  • Dependencies: none except for the build environment
  • Protocol Identification:
    • Payload Matches
    • Payload Size
    • Port Number
    • IP Matching
    • ability to specify custom ports for protocol in specific environment through a configuration file
  • Multi thread support: yes
  • Flow-aware: yes (embedded)
  • Test Pcap: few in the code base

Community support

  • libprotoident
    The source code is openly available on GitHub, at the moment there is only one author active.
  • nDPI
    The source code is openly available on GitHub.

Licence

  • libprotoident
    GPLv2.

Supported Protocols

Interesting Papers

Actions #7

Updated by Vito Piserchia over 5 years ago

Important statement about the libprotoident and its future from his author can be found here: https://github.com/wanduow/libprotoident/issues/12

Actions #8

Updated by Andreas Herz over 5 years ago

Does anyone have some real experience with those projects and could share that knowledge?
Might be worth to take a look at it but seems like a more time consuming task :)

Actions #9

Updated by Victor Julien over 5 years ago

  • Assignee changed from Victor Julien to Anonymous

I think we can rule out nDPI for the licensing issue. The libprotoident might be worth looking into, although I'm a bit worried about it's continued development. Also the further dependencies the lib has might be an issue.

Actions #10

Updated by Vito Piserchia over 5 years ago

The libprotoident comes out with a few examples, in the tools folder of the code. From the README https://github.com/wanduow/libprotoident

  • lpi_protoident

    Description:
    This tool attempts to identify each individual flow within the provided
    trace. Identification only occurs when the flow has concluded or
    expired, so it is not very effective for real-time applications.

  • lpi_live (DEPRECATED)

    Description:
    This tool reports byte and packet counts (both inbound and outbound)
    for each identified protocol in real-time. Identification of a flow
    occurs as soon as possible, so that the statistics reported are as
    up-to-date as possible.
    lpi_live has been deprecated and will not be built by default. The code
    is still available in our git repository, but we will not update or
    support this tool anymore. Instead, please use the lpicollector
    (https://github.com/wanduow/lpicollector) for real-time flow analysis
    with libprotoident.
    In combination with the included lpi.py example client, lpicollector
    can produce output similar to that produced by lpi_live.

Libprotoident calls (to external libs) and mappings to suricata's (tentative)

from libtrace:

  • types

libtrace_t | NOT NEEDED
libtrace_tcp_t | combination of PKT_IS_TCP(p) && (p)->tcph
libtrace_udp_t | combination of PKT_IS_UDP(p) && (p)->udph
libtrace_packet_t | analogous to the Packet struct
libtrace_filter_t | use the suricata's buildin BPF code

  • calls

trace_get_layer3 | IP_GET_IPPROTO(p)
trace_get_direction | FlowGetPacketDirection(f, p) == TOSERVER ? 1 : 0; NOTE: they are reversed
trace_get_payload | (Packet *)p->payload
trace_get_payload_length | (Packet *)p->payload_len
trace_read_packet | NOT NEEDED, use suricata source modules
trace_get_seconds | XXX
trace_create | NOT NEEDED, use suricata source modules
trace_destroy | NOT NEEDED, use suricata source modules
trace_create_filter | use suricata's BPF filtering
trace_perror | NOT NEEDED
trace_is_err | NOT NEEDED

from libflowmanager

NEEDED | lfm_match_packet_to_flow | FlowReference(&p->flow, f);
NEEDED | lfm_update_flow_expiry_timeout | this is done via the flow-timeouts values
NOT NEEDED | lfm_expire_next_flow
NOT NEEDED | lfm_set_config_option

Actions #11

Updated by Fanny Dwargee about 4 years ago

Do we have any progress with this feature?

It seems that the GitHub repository of libprotoident is updated frequently (at least in the last months) and this would make Suricata another step ahead of Snort :)

Actions #12

Updated by Victor Julien almost 3 years ago

Actions #13

Updated by Vito Piserchia almost 3 years ago

Since from 6th August 2016 libprotoident is GNUv3 licensed [1], putting it in the same situation as nDPI

There is an open ticket [2] but it got no reply since Mar 9, 2017

[1] https://github.com/wanduow/libprotoident/commit/de8e2ca6d6eb04526912dbc433b0c6003b9f65e1#diff-7116ef0705885343c9e1b2171a06be0e
[2] https://github.com/wanduow/libprotoident/issues/20

Actions #14

Updated by Andreas Herz over 2 years ago

  • Assignee set to Community Ticket
Actions

Also available in: Atom PDF