https://redmine.openinfosecfoundation.org/https://redmine.openinfosecfoundation.org/favicon.ico?17011170022012-08-06T10:08:57ZOpen Information Security FoundationSuricata - Feature #511: Port indepedent protocol identification (nDPI)https://redmine.openinfosecfoundation.org/issues/511?journal_id=19962012-08-06T10:08:57ZVictor Julienvictor@inliniac.net
<ul><li><strong>Target version</strong> set to <i>TBD</i></li></ul><p>OpenDPI/nDPI is licensed LGPLv3, Suricata is GPLv2. According to <a class="external" href="http://www.gnu.org/licenses/gpl-faq.html#AllCompatibility">http://www.gnu.org/licenses/gpl-faq.html#AllCompatibility</a> these are incompatible.</p> Suricata - Feature #511: Port indepedent protocol identification (nDPI)https://redmine.openinfosecfoundation.org/issues/511?journal_id=21082012-08-27T02:20:31ZVictor Julienvictor@inliniac.net
<ul><li><strong>Assignee</strong> set to <i>Victor Julien</i></li></ul> Suricata - Feature #511: Port indepedent protocol identification (nDPI)https://redmine.openinfosecfoundation.org/issues/511?journal_id=21592012-09-06T10:11:01ZVictor Julienvictor@inliniac.net
<ul></ul><p>In the process of asking for legal advice on this.</p> Suricata - Feature #511: Port indepedent protocol identification (nDPI)https://redmine.openinfosecfoundation.org/issues/511?journal_id=21602012-09-08T15:48:29ZVictor Julienvictor@inliniac.net
<ul></ul><p>Another lib: <a class="external" href="http://research.wand.net.nz/software/libprotoident.php">http://research.wand.net.nz/software/libprotoident.php</a></p> Suricata - Feature #511: Port indepedent protocol identification (nDPI)https://redmine.openinfosecfoundation.org/issues/511?journal_id=58492015-12-22T16:02:46ZAndreas Herzoisf@herzandreas.de
<ul></ul><p>Any update on the legal issue? the lib from wand looks not really up to date</p> Suricata - Feature #511: Port indepedent protocol identification (nDPI)https://redmine.openinfosecfoundation.org/issues/511?journal_id=67272016-04-28T06:18:44ZVito Piserchia
<ul></ul><p>I'll try to put and add more information about different protocol inspection projects, for now:</p>
<ul>
<li>libprotoident</li>
<li>nDPI</li>
</ul>
<a name="Generalities-contact-page-main-development-site-etc-etc"></a>
<h1 >Generalities (contact page, main development site, etc etc)<a href="#Generalities-contact-page-main-development-site-etc-etc" class="wiki-anchor">¶</a></h1>
<a name="libprotoident"></a>
<h2 ><strong>libprotoident</strong><a href="#libprotoident" class="wiki-anchor">¶</a></h2>
<a name="Description"></a>
<h3 >Description:<a href="#Description" class="wiki-anchor">¶</a></h3>
<p>(from the project page)</p>
<p>Libprotoident is a library that performs application layer protocol identification for flows. Unlike many techniques that require capturing the entire packet payload, only the <strong>first four bytes</strong> of payload sent in each direction, the size of the first payload-bearing packet in each direction and the TCP or UDP port numbers for the flow are used by libprotoident.</p>
<ul>
<li>project page: <a class="external" href="http://research.wand.net.nz/software/libprotoident.php">http://research.wand.net.nz/software/libprotoident.php</a></li>
<li>source code: <a class="external" href="https://github.com/wanduow/libprotoident">https://github.com/wanduow/libprotoident</a></li>
<li>blog and news: <a class="external" href="https://secure.wand.net.nz/projects/details/libprotoident">https://secure.wand.net.nz/projects/details/libprotoident</a></li>
<li>wiki: <a class="external" href="https://secure.wand.net.nz/trac/libprotoident/">https://secure.wand.net.nz/trac/libprotoident/</a></li>
</ul>
<a name="nDPI"></a>
<h2 ><strong>nDPI</strong><a href="#nDPI" class="wiki-anchor">¶</a></h2>
<a name="Description-2"></a>
<h3 >Description<a href="#Description-2" class="wiki-anchor">¶</a></h3>
<p>(from the project page):</p>
<p>nDPI is a ntop-maintained superset of the popular OpenDPI library. Released under the LGPL license, its goal is to extend the original library by adding new protocols that are otherwise available only on the paid version of OpenDPI. In addition to Unix platforms, we also support Windows, in order to provide you a cross-platform DPI experience. Furthermore, we have modified nDPI do be more suitable for traffic monitoring applications, by disabling specific features that slow down the DPI engine while being them un-necessary for network traffic monitoring.</p>
<ul>
<li>project page: <a class="external" href="http://www.ntop.org/products/deep-packet-inspection/ndpi/">http://www.ntop.org/products/deep-packet-inspection/ndpi/</a></li>
<li>source code: <a class="external" href="https://github.com/ntop/nDPI">https://github.com/ntop/nDPI</a></li>
<li>blog and news: <a class="external" href="http://ntop.org">http://ntop.org</a></li>
</ul>
<a name="Specificities"></a>
<h2 > Specificities<a href="#Specificities" class="wiki-anchor">¶</a></h2>
<a name="libprotoident-2"></a>
<h3 >libprotoident<a href="#libprotoident-2" class="wiki-anchor">¶</a></h3>
<ul>
<li>Language: C++</li>
</ul>
<ul>
<li>Dependencies: wand <strong>libtrace</strong>, from <a class="external" href="http://research.wand.net.nz/software/libtrace.php">http://research.wand.net.nz/software/libtrace.php</a></li>
</ul>
<ul>
<li>Protocol Identification:
<ul>
<li>Payload Matches (mostly), pattern matching on the first four bytes of the payload on each direction of the traffic</li>
<li>Payload Size</li>
<li>Port Number, used in case of ambiguity and only for well known ports</li>
<li>IP Matching, very few cases </li>
<li>protocols are checked in order depending on the confidence the author have on the rules and the popularity of the protocol</li>
</ul></li>
</ul>
<ul>
<li>Multi thread support: no, but can be added in the user application</li>
</ul>
<ul>
<li>Flow-aware: no, but can use the wand <strong>libflowmanager</strong>, from <a class="external" href="http://research.wand.net.nz/software/libflowmanager.php">http://research.wand.net.nz/software/libflowmanager.php</a>. Only needed for building the tools</li>
</ul>
<ul>
<li>Test Pcap: none</li>
</ul>
<a name="nDPI-2"></a>
<h3 >nDPI<a href="#nDPI-2" class="wiki-anchor">¶</a></h3>
<ul>
<li>Language: C</li>
</ul>
<ul>
<li>Dependencies: none except for the build environment</li>
</ul>
<ul>
<li>Protocol Identification:
<ul>
<li>Payload Matches</li>
<li>Payload Size</li>
<li>Port Number</li>
<li>IP Matching</li>
<li>ability to specify custom ports for protocol in specific environment through a configuration file</li>
</ul></li>
</ul>
<ul>
<li>Multi thread support: yes</li>
</ul>
<ul>
<li>Flow-aware: yes (embedded)</li>
</ul>
<ul>
<li>Test Pcap: few in the code base</li>
</ul>
<a name="Community-support"></a>
<h3 >Community support<a href="#Community-support" class="wiki-anchor">¶</a></h3>
<ul>
<li>libprotoident<br />The source code is openly available on GitHub, at the moment there is only one author active.</li>
</ul>
<ul>
<li>nDPI<br />The source code is openly available on GitHub.</li>
</ul>
<a name="Licence"></a>
<h3 >Licence<a href="#Licence" class="wiki-anchor">¶</a></h3>
<ul>
<li>libprotoident<br />GPLv2.</li>
</ul>
<ul>
<li>nDPI<br />LGPLv3. This is an issue if you want to include its source into suricat, according to <a class="external" href="http://www.gnu.org/licenses/gpl-faq.html#AllCompatibility">http://www.gnu.org/licenses/gpl-faq.html#AllCompatibility</a></li>
</ul>
<a name="Supported-Protocols"></a>
<h3 >Supported Protocols<a href="#Supported-Protocols" class="wiki-anchor">¶</a></h3>
<ul>
<li>libprotoident<br />A list is here (updated?) : <a class="external" href="https://secure.wand.net.nz/trac/libprotoident/wiki/SupportedProtocols">https://secure.wand.net.nz/trac/libprotoident/wiki/SupportedProtocols</a></li>
</ul>
<ul>
<li>nDPI<br />A list is present on the main project page: <a class="external" href="http://www.ntop.org/products/deep-packet-inspection/ndpi/">http://www.ntop.org/products/deep-packet-inspection/ndpi/</a></li>
</ul>
<a name="Interesting-Papers"></a>
<h2 >Interesting Papers<a href="#Interesting-Papers" class="wiki-anchor">¶</a></h2>
<ul>
<li>T. Bujlow et al, Jan. 2014 - Extended Independent Comparison of Popular Deep Packet Inspection (DPI) Tools for Traffic Classification (<a class="external" href="http://www.ac.upc.edu/app/research-reports/html/RR/2014/1.pdf">http://www.ac.upc.edu/app/research-reports/html/RR/2014/1.pdf</a>) <br />Very long (more than 400 pages) comparison of the most popular and public available DPI engine. A shorter article version from the same author also exists (Nov. 2014, <a class="external" href="http://tomasz.bujlow.com/publications/2014_journal_elsevier_comnet_independent_comparison.pdf">http://tomasz.bujlow.com/publications/2014_journal_elsevier_comnet_independent_comparison.pdf</a>)</li>
</ul> Suricata - Feature #511: Port indepedent protocol identification (nDPI)https://redmine.openinfosecfoundation.org/issues/511?journal_id=67282016-04-29T02:45:03ZVito Piserchia
<ul></ul><p>Important statement about the <strong>libprotoident</strong> and its future from his author can be found here: <a class="external" href="https://github.com/wanduow/libprotoident/issues/12">https://github.com/wanduow/libprotoident/issues/12</a></p> Suricata - Feature #511: Port indepedent protocol identification (nDPI)https://redmine.openinfosecfoundation.org/issues/511?journal_id=67302016-05-02T16:58:15ZAndreas Herzoisf@herzandreas.de
<ul></ul><p>Does anyone have some real experience with those projects and could share that knowledge?<br />Might be worth to take a look at it but seems like a more time consuming task :)</p> Suricata - Feature #511: Port indepedent protocol identification (nDPI)https://redmine.openinfosecfoundation.org/issues/511?journal_id=67322016-05-03T09:08:44ZVictor Julienvictor@inliniac.net
<ul><li><strong>Assignee</strong> changed from <i>Victor Julien</i> to <i>Anonymous</i></li></ul><p>I think we can rule out nDPI for the licensing issue. The libprotoident might be worth looking into, although I'm a bit worried about it's continued development. Also the further dependencies the lib has might be an issue.</p> Suricata - Feature #511: Port indepedent protocol identification (nDPI)https://redmine.openinfosecfoundation.org/issues/511?journal_id=67392016-05-03T10:15:14ZVito Piserchia
<ul></ul><p>The libprotoident comes out with a few examples, in the tools folder of the code. From the README <a class="external" href="https://github.com/wanduow/libprotoident">https://github.com/wanduow/libprotoident</a></p>
<ul>
<li>lpi_protoident
<p>Description:<br /> This tool attempts to identify each individual flow within the provided<br /> trace. Identification only occurs when the flow has concluded or <br /> expired, so it is not very effective for real-time applications.</p></li>
</ul>
<ul>
<li>lpi_live (DEPRECATED)
<p>Description:<br /> This tool reports byte and packet counts (both inbound and outbound)<br /> for each identified protocol in real-time. Identification of a flow<br /> occurs as soon as possible, so that the statistics reported are as<br /> up-to-date as possible.<br />lpi_live has been deprecated and will not be built by default. The code<br /> is still available in our git repository, but we will not update or<br /> support this tool anymore. Instead, please use the lpicollector<br /> (<a class="external" href="https://github.com/wanduow/lpicollector">https://github.com/wanduow/lpicollector</a>) for real-time flow analysis<br /> with libprotoident.<br /> In combination with the included lpi.py example client, lpicollector<br /> can produce output similar to that produced by lpi_live.</p></li>
</ul>
<a name="Libprotoident-calls-to-external-libs-and-mappings-to-suricatas-tentative"></a>
<h2 >Libprotoident calls (to external libs) and mappings to suricata's (tentative)<a href="#Libprotoident-calls-to-external-libs-and-mappings-to-suricatas-tentative" class="wiki-anchor">¶</a></h2>
<a name="from-libtrace"></a>
<h3 >from libtrace:<a href="#from-libtrace" class="wiki-anchor">¶</a></h3>
<ul>
<li>types</li>
</ul>
<p>libtrace_t | NOT NEEDED<br />libtrace_tcp_t | combination of PKT_IS_TCP(p) && (p)->tcph <br />libtrace_udp_t | combination of PKT_IS_UDP(p) && (p)->udph<br />libtrace_packet_t | analogous to the <strong>Packet</strong> struct<br />libtrace_filter_t | use the suricata's buildin BPF code</p>
<ul>
<li>calls</li>
</ul>
<p>trace_get_layer3 | IP_GET_IPPROTO(p)<br />trace_get_direction | FlowGetPacketDirection(f, p) == TOSERVER ? 1 : 0; <strong>NOTE</strong>: they are reversed<br />trace_get_payload | (Packet *)p->payload <br />trace_get_payload_length | (Packet *)p->payload_len<br />trace_read_packet | NOT NEEDED, use suricata source modules<br />trace_get_seconds | XXX<br />trace_create | NOT NEEDED, use suricata source modules<br />trace_destroy | NOT NEEDED, use suricata source modules<br />trace_create_filter | use suricata's BPF filtering<br />trace_perror | NOT NEEDED<br />trace_is_err | NOT NEEDED</p>
<a name="from-libflowmanager"></a>
<h3 >from libflowmanager<a href="#from-libflowmanager" class="wiki-anchor">¶</a></h3>
<p>NEEDED | lfm_match_packet_to_flow | FlowReference(&p->flow, f); <br />NEEDED | lfm_update_flow_expiry_timeout | this is done via the flow-timeouts values<br />NOT NEEDED | lfm_expire_next_flow<br />NOT NEEDED | lfm_set_config_option</p> Suricata - Feature #511: Port indepedent protocol identification (nDPI)https://redmine.openinfosecfoundation.org/issues/511?journal_id=84842017-07-13T05:14:27ZFanny Dwargee
<ul></ul><p>Do we have any progress with this feature?</p>
<p>It seems that the GitHub repository of libprotoident is updated frequently (at least in the last months) and this would make Suricata another step ahead of Snort :)</p> Suricata - Feature #511: Port indepedent protocol identification (nDPI)https://redmine.openinfosecfoundation.org/issues/511?journal_id=107292018-12-21T12:07:13ZVictor Julienvictor@inliniac.net
<ul><li><strong>Related to</strong> <i><a class="issue tracker-2 status-7 priority-4 priority-default" href="/issues/2757">Feature #2757</a>: improve protocol detection</i> added</li></ul> Suricata - Feature #511: Port indepedent protocol identification (nDPI)https://redmine.openinfosecfoundation.org/issues/511?journal_id=107442018-12-21T18:23:28ZVito Piserchia
<ul></ul><p>Since from 6th August 2016 libprotoident is GNUv3 licensed [1], putting it in the same situation as nDPI</p>
<p>There is an open ticket [2] but it got no reply since Mar 9, 2017</p>
<p>[1] <a class="external" href="https://github.com/wanduow/libprotoident/commit/de8e2ca6d6eb04526912dbc433b0c6003b9f65e1#diff-7116ef0705885343c9e1b2171a06be0e">https://github.com/wanduow/libprotoident/commit/de8e2ca6d6eb04526912dbc433b0c6003b9f65e1#diff-7116ef0705885343c9e1b2171a06be0e</a><br />[2] <a class="external" href="https://github.com/wanduow/libprotoident/issues/20">https://github.com/wanduow/libprotoident/issues/20</a></p> Suricata - Feature #511: Port indepedent protocol identification (nDPI)https://redmine.openinfosecfoundation.org/issues/511?journal_id=111642019-02-23T22:09:34ZAndreas Herzoisf@herzandreas.de
<ul><li><strong>Assignee</strong> set to <i>Community Ticket</i></li></ul> Suricata - Feature #511: Port indepedent protocol identification (nDPI)https://redmine.openinfosecfoundation.org/issues/511?journal_id=292622023-07-19T14:53:23ZPhilippe Antoine
<ul></ul><p><a class="user active user-mention" href="https://redmine.openinfosecfoundation.org/users/9">@Jason Ish</a> could this be a use case for a plugin ?</p> Suricata - Feature #511: Port indepedent protocol identification (nDPI)https://redmine.openinfosecfoundation.org/issues/511?journal_id=293122023-07-21T08:50:41ZPhilippe Antoine
<ul><li><strong>Subject</strong> changed from <i>Port indepedent protocol identification</i> to <i>Port indepedent protocol identification (nDPI)</i></li></ul> Suricata - Feature #511: Port indepedent protocol identification (nDPI)https://redmine.openinfosecfoundation.org/issues/511?journal_id=294062023-07-31T22:08:47ZJason Ishjason.ish@oisf.net
<ul></ul><p>Philippe Antoine wrote in <a href="#note-15">#note-15</a>:</p>
<blockquote>
<p><a class="user active user-mention" href="https://redmine.openinfosecfoundation.org/users/9">@Jason Ish</a> could this be a use case for a plugin ?</p>
</blockquote>
<p>Not sure.. This sounds like something that's pretty core? Did you have an idea in mind how it might make sense as a plugin?</p> Suricata - Feature #511: Port indepedent protocol identification (nDPI)https://redmine.openinfosecfoundation.org/issues/511?journal_id=305672023-11-07T18:33:04ZBrandon Murphy
<ul></ul><p>just adding a +1 to this feature for nDPI. When Victor was speaking during the "State of Suricata" about plugins, nDPI was the first one that came to my mind. I've talked to a lot of users who desire this type of feature within suricata.</p>