Project

General

Profile

Actions

Bug #3096

closed

random failures on sip and http-evader suricata-verify tests

Added by Victor Julien over 3 years ago. Updated almost 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:
Needs backport

Description

every couple of travis builds there seems to be a random failure in one of the http-evader tests that goes away by simply restarting the job.


Related issues 2 (0 open2 closed)

Copied to Bug #3581: random failures on sip and http-evader suricata-verify testsClosedShivani BhardwajActions
Copied to Bug #3582: random failures on sip and http-evader suricata-verify testsClosedJeff LucovskyActions
Actions #1

Updated by Victor Julien over 3 years ago

  • Status changed from New to Assigned
  • Assignee set to Philippe Antoine
  • Target version set to TBD

Philippe, can you see if this is something specific to the evader tests and if it can be reproduced locally?

Actions #2

Updated by Philippe Antoine over 3 years ago

Can you point me to one failed Travis build because of this ?
I did not find one in https://travis-ci.org/OISF/suricata/builds

Actions #3

Updated by Philippe Antoine over 3 years ago

Hmm... I got one for http-evader-306
https://travis-ci.org/catenacyber/suricata/jobs/563954182

===> http-evader-305: OK
===> http-evader-306: Sub test #1: FAIL : expected 1 matches; got 0 for filter {'count': 1, 'match': {'alert.signature_id': 1, 'event_type': 'alert'}}
Sub test #2: FAIL : expected 1 matches; got 0 for filter {'count': 1, 'match': {'alert.signature_id': 2, 'event_type': 'alert'}}
Sub test #3: FAIL : expected 1 matches; got 0 for filter {'count': 1, 'match': {'http.hostname': 'evader.example.com', 'http.http_method': 'GET', 'event_type': 'fileinfo', 'fileinfo.filename': 'eicar.txt'}}
===> http-evader-307: OK

Actions #4

Updated by Philippe Antoine about 3 years ago

This might be related :
https://travis-ci.org/OISF/suricata/jobs/593933093

===> show-help: OK
===> sip-method: Sub test #1: FAIL : expected 36 matches; got 35 for filter {'count': 36, 'match': {'event_type': 'alert'}}
===> sip-protocol: OK

That is not http evader, but maybe this is a bug of suricata-verify, and that comes more often with http-evader tests since these are the majority of the run tests

Actions #5

Updated by Victor Julien about 3 years ago

Maybe, but in my own QA I see the sip failures a lot, http-evader failures seldom, and no other test fail. So I'm inclined to think the sip tests (or suricata sip code) has a different issue.

Actions #6

Updated by Philippe Antoine about 3 years ago

Victor, could you tell me which http-evader tests fail in your own QA ?

Actions #7

Updated by Victor Julien about 3 years ago

I haven't kept a list. Will make a note for future failures.

Actions #8

Updated by Philippe Antoine about 3 years ago

Got a new one : https://travis-ci.org/catenacyber/suricata/jobs/604848050?utm_medium=notification&utm_source=email

===> http-evader-492: OK
3204===> http-evader-493: Sub test #1: FAIL : expected 1 matches; got 0 for filter {'count': 1, 'match': {'alert.signature_id': 1, 'event_type': 'alert'}}
3205Sub test #2: FAIL : expected 1 matches; got 0 for filter {'count': 1, 'match': {'alert.signature_id': 2, 'event_type': 'alert'}}
3206Sub test #3: FAIL : expected 1 matches; got 0 for filter {'count': 1, 'match': {'http.hostname': 'evader.example.com', 'http.http_method': 'GET', 'event_type': 'fileinfo', 'fileinfo.filename': 'eicar.txt'}}
3207===> http-evader-494: OK

Actions #9

Updated by Philippe Antoine about 3 years ago

Should we get stdout/stderr from failed run tests in Travis ?
So we can debug this flaky behavior...

Actions #10

Updated by Philippe Antoine almost 3 years ago

  • Status changed from Assigned to In Review
Actions #11

Updated by Victor Julien almost 3 years ago

I'm seeing these issues a lot less frequent. In fact, can't remember the last time one of the http tests failed. I do occasionally see one of the sip tests fail, (almost?) always on arm64 it seems.

Actions #12

Updated by Philippe Antoine almost 3 years ago

I did not see a http test fail recently.

Though, some random failure will happen some time and we will want more debug information.
So https://github.com/OISF/suricata-verify/pull/161 is still useful even if this redmine ticket title is not entirely accurate

Actions #13

Updated by Victor Julien almost 3 years ago

Philippe Antoine wrote in #note-12:

Though, some random failure will happen some time and we will want more debug information.
So https://github.com/OISF/suricata-verify/pull/161 is still useful even if this redmine ticket title is not entirely accurate

Agreed.

Actions #14

Updated by Victor Julien almost 3 years ago

  • Status changed from In Review to Assigned
  • Assignee changed from Philippe Antoine to Victor Julien
  • Priority changed from Normal to High
  • Target version changed from TBD to 6.0.0beta1
  • Label Needs backport added

I've identified a corner case where at the start of processing a pcap, flows may get evicted from the flow hash too early. On slower hardware this is more likely to happen. I've been able to fairly reliably reproduce this with the sip tests on a small arm64 device. Working on a fix.

Actions #15

Updated by Victor Julien almost 3 years ago

  • Subject changed from random failures on http-evader suricata-verify tests to random failures on sip and http-evader suricata-verify tests
Actions #16

Updated by Victor Julien almost 3 years ago

  • Status changed from Assigned to Closed
  • Priority changed from High to Normal
Actions #17

Updated by Jeff Lucovsky over 2 years ago

  • Copied to Bug #3581: random failures on sip and http-evader suricata-verify tests added
Actions #18

Updated by Jeff Lucovsky over 2 years ago

  • Copied to Bug #3582: random failures on sip and http-evader suricata-verify tests added
Actions

Also available in: Atom PDF