https://redmine.openinfosecfoundation.org/https://redmine.openinfosecfoundation.org/favicon.ico?17011170022019-11-05T12:07:30ZOpen Information Security FoundationSuricata - Feature #3317: rules: use rust for tokenizing ruleshttps://redmine.openinfosecfoundation.org/issues/3317?journal_id=145362019-11-05T12:07:30ZVictor Julienvictor@inliniac.net
<ul><li><strong>Related to</strong> <i><a class="issue tracker-5 status-1 priority-5 priority-high3" href="/issues/3195">Task #3195</a>: tracking: rustify all input</i> added</li></ul> Suricata - Feature #3317: rules: use rust for tokenizing ruleshttps://redmine.openinfosecfoundation.org/issues/3317?journal_id=145372019-11-05T12:11:18ZVictor Julienvictor@inliniac.net
<ul><li><strong>Subject</strong> changed from <i>rules: use rule for tokenizing rules</i> to <i>rules: use rust for tokenizing rules</i></li></ul> Suricata - Feature #3317: rules: use rust for tokenizing ruleshttps://redmine.openinfosecfoundation.org/issues/3317?journal_id=145392019-11-05T12:40:43ZVictor Julienvictor@inliniac.net
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-1 priority-4 priority-default child" href="/issues/1926">Bug #1926</a>: rule parsing: wrong content checked for fast_pattern (snort compatibility)</i> added</li></ul> Suricata - Feature #3317: rules: use rust for tokenizing ruleshttps://redmine.openinfosecfoundation.org/issues/3317?journal_id=145672019-11-06T11:31:11ZVictor Julienvictor@inliniac.net
<ul></ul><p>It would be nice if this could replace much of our pcre use in parsing. The pcre use is both for tokenizing and input validation. The tokenizing works well, but the input validation less so. Its hard to produce clear errors that are better than "the regex said no".</p>
<p>If the rust based code does all the tokenizing, we'd need more input value validation to make up for it.</p> Suricata - Feature #3317: rules: use rust for tokenizing ruleshttps://redmine.openinfosecfoundation.org/issues/3317?journal_id=145682019-11-06T11:31:57ZVictor Julienvictor@inliniac.net
<ul></ul><p>(by Jason Ish. Moved from <a class="issue tracker-5 status-1 priority-5 priority-high3" title="Task: tracking: rustify all input (New)" href="https://redmine.openinfosecfoundation.org/issues/3195">#3195</a>)</p>
<p>Victor Julien wrote:</p>
<blockquote>
<p>It would be nice if this could replace much of our pcre use in parsing. The pcre use is both for tokenizing and input validation. The tokenizing works well, but the input validation less so. Its hard to produce clear errors that are better than "the regex said no".</p>
<p>If the rust based code does all the tokenizing, we'd need more input value validation to make up for it.</p>
</blockquote>
<p>The way I see is it the top level parser will give you a tuple of (keyword, value), but will not have done any validation of that value to make sure its correct for that keyword. It will be up to the handler for that keyword to parse that like it is now. So it would get rid of pcre in this outer tokenizer, but the parsing of the values would be on a keyword by keyword basis.</p> Suricata - Feature #3317: rules: use rust for tokenizing ruleshttps://redmine.openinfosecfoundation.org/issues/3317?journal_id=145692019-11-06T11:35:03ZVictor Julienvictor@inliniac.net
<ul></ul><p>I think this highest level tokenizing is already done w/o pcre since some time.</p>
<p>Not sure I see much value in a rust crate that would just do the highest level of tokenizing. I was more thinking about having a rule parser that could be the single source of 'truth' for Suricata rule parsing and validation. This mean it would have to be much more aware of the individual keywords and their syntax. Maybe this isn't feasible.</p> Suricata - Feature #3317: rules: use rust for tokenizing ruleshttps://redmine.openinfosecfoundation.org/issues/3317?journal_id=145712019-11-06T14:06:46ZJason Ishjason.ish@oisf.net
<ul></ul><p>Its a mix of effort and re-usability I think.</p>
<p>A tokenizer would satisfy requirements of preprocessing rules - such as Suricata-Update, or basic enable/disables.</p>
<p>You typically might tokenize, then pass off to the parser. We could implement that as a standalone module, but is a lot more work as it has to understand every keyword. You'd want to parse all the values into some struct that could then be use by that keyword implementation, so it doesn't need to reparse the value again. We could have a generic one for keywords that are unknown to the parser, which would allow us to implement keywords over time.</p>
<p>But I see it as a tokenizer, which would then feed to a parser.</p> Suricata - Feature #3317: rules: use rust for tokenizing ruleshttps://redmine.openinfosecfoundation.org/issues/3317?journal_id=145762019-11-06T18:54:05ZVictor Julienvictor@inliniac.net
<ul></ul><p>Ok, I guess I see little to no value in just a simple high level tokenizer in Rust. The current code is fast and simple, so we wouldn't gain much.</p> Suricata - Feature #3317: rules: use rust for tokenizing ruleshttps://redmine.openinfosecfoundation.org/issues/3317?journal_id=181262020-11-02T08:01:27ZVictor Julienvictor@inliniac.net
<ul><li><strong>Related to</strong> <i><a class="issue tracker-5 status-1 priority-4 priority-default parent" href="/issues/4095">Task #4095</a>: tracking: unify rule keyword value parsing</i> added</li></ul> Suricata - Feature #3317: rules: use rust for tokenizing ruleshttps://redmine.openinfosecfoundation.org/issues/3317?journal_id=214622021-11-26T13:21:13ZVictor Julienvictor@inliniac.net
<ul><li><strong>Parent task</strong> set to <i>#4855</i></li></ul>