Project

General

Profile

Feature #3082

suricata-verify: option to test 'skipped' tests

Added by Victor Julien 3 months ago. Updated about 1 month ago.

Status:
Assigned
Priority:
Normal
Target version:
Effort:
medium
Difficulty:
low
Label:

Description

Now that the number of tests is growing a new issue appears: which of the http evader tests are supposed to no longer be skipped after the last merges? We can try declaring HTTP_EVADER_OK and see which tests pass, but then we'd have to correlate that with the list of skipped tests if HTTP_EVADER_OK is not declared. Quite cumbersome.

So my suggestion is to add a way to check this. Run suricata-verify with an option to try tests with a skip condition anyway.

Something like:

run.py --test-skip HTTP_EVADER_OK

This would then only run those tests that have HTTP_EVADER_OK in their requires and would output something like:
===> http-evader-447: FAILED requires HTTP_EVADER_OK, so considering OK
===> http-evader-448: OK - is requires HTTP_EVADER_OK needed?
..
PASSED:  1368
FAILED:  0
SKIPPED: 149
SKIPPED OK: 1
SKIPPED FAIL: 1

This would then make it clear which of the tests needs to be updated.


Related issues

Related to Optimization #3172: Give only test count as outputFeedbackActions

History

#1

Updated by Victor Julien 3 months ago

  • Target version set to QA
#2

Updated by Andreas Herz 3 months ago

  • Assignee set to OISF Dev
  • Effort set to medium
  • Difficulty set to low
#3

Updated by Shivani Bhardwaj 3 months ago

  • Status changed from New to Assigned
  • Assignee changed from OISF Dev to Shivani Bhardwaj
#4

Updated by Peter Manev 3 months ago

off note - i just saw a small miss spell

 http-evader-448: OK - is requires ...

"is requires" should probably be "required".

(newbie :) ) Question - do we have a way to list what can be declared to be skipped ? Or how to list multiples tests sequences to be skipped?

#5

Updated by Jason Ish 3 months ago

Related: You can tell suricata-verify what tests to run as well, for example:

.../run.py dns

will only run tests with "dns" in the name.

.../run.py dns http

will run tests with "dns" OR "http" in the name.

#6

Updated by Shivani Bhardwaj 2 months ago

  • Status changed from Assigned to Feedback
#7

Updated by Shivani Bhardwaj about 1 month ago

  • Status changed from Feedback to Assigned
#8

Updated by Shivani Bhardwaj about 14 hours ago

Also available in: Atom PDF