Suricata

Can you point me to one failed Travis build because of this ?
I did not find one in https://travis-ci.org/OISF/suricata/builds

Hmm... I got one for http-evader-306
https://travis-ci.org/catenacyber/suricata/jobs/563954182

===> http-evader-305: OK
===> http-evader-306: Sub test #1: FAIL : expected 1 matches; got 0 for filter {'count': 1, 'match': {'alert.signature_id': 1, 'event_type': 'alert'}}
Sub test #2: FAIL : expected 1 matches; got 0 for filter {'count': 1, 'match': {'alert.signature_id': 2, 'event_type': 'alert'}}
Sub test #3: FAIL : expected 1 matches; got 0 for filter {'count': 1, 'match': {'http.hostname': 'evader.example.com', 'http.http_method': 'GET', 'event_type': 'fileinfo', 'fileinfo.filename': 'eicar.txt'}}
===> http-evader-307: OK

This might be related :
https://travis-ci.org/OISF/suricata/jobs/593933093

===> show-help: OK
===> sip-method: Sub test #1: FAIL : expected 36 matches; got 35 for filter {'count': 36, 'match': {'event_type': 'alert'}}
===> sip-protocol: OK

That is not http evader, but maybe this is a bug of suricata-verify, and that comes more often with http-evader tests since these are the majority of the run tests

Maybe, but in my own QA I see the sip failures a lot, http-evader failures seldom, and no other test fail. So I'm inclined to think the sip tests (or suricata sip code) has a different issue.

Victor, could you tell me which http-evader tests fail in your own QA ?

I haven't kept a list. Will make a note for future failures.

Got a new one : https://travis-ci.org/catenacyber/suricata/jobs/604848050?utm_medium=notification&utm_source=email

===> http-evader-492: OK
3204===> http-evader-493: Sub test #1: FAIL : expected 1 matches; got 0 for filter {'count': 1, 'match': {'alert.signature_id': 1, 'event_type': 'alert'}}
3205Sub test #2: FAIL : expected 1 matches; got 0 for filter {'count': 1, 'match': {'alert.signature_id': 2, 'event_type': 'alert'}}
3206Sub test #3: FAIL : expected 1 matches; got 0 for filter {'count': 1, 'match': {'http.hostname': 'evader.example.com', 'http.http_method': 'GET', 'event_type': 'fileinfo', 'fileinfo.filename': 'eicar.txt'}}
3207===> http-evader-494: OK

Should we get stdout/stderr from failed run tests in Travis ?
So we can debug this flaky behavior...

I'm seeing these issues a lot less frequent. In fact, can't remember the last time one of the http tests failed. I do occasionally see one of the sip tests fail, (almost?) always on arm64 it seems.

I did not see a http test fail recently.

Though, some random failure will happen some time and we will want more debug information.
So https://github.com/OISF/suricata-verify/pull/161 is still useful even if this redmine ticket title is not entirely accurate

Philippe Antoine wrote in #note-12:

Though, some random failure will happen some time and we will want more debug information.
So https://github.com/OISF/suricata-verify/pull/161 is still useful even if this redmine ticket title is not entirely accurate

Agreed.