Project

General

Profile

Actions
Copy link

Feature #3082

closed
VJ

suricata-verify: option to test 'skipped' tests

Feature #3082: suricata-verify: option to test 'skipped' tests

Added by Victor Julien almost 7 years ago. Updated about 1 month ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
QA
Effort:
Difficulty:
Label:
Python

Description

Now that the number of tests is growing a new issue appears: which of the http evader tests are supposed to no longer be skipped after the last merges? We can try declaring HTTP_EVADER_OK and see which tests pass, but then we'd have to correlate that with the list of skipped tests if HTTP_EVADER_OK is not declared. Quite cumbersome.

So my suggestion is to add a way to check this. Run suricata-verify with an option to try tests with a skip condition anyway.

Something like:

run.py --test-skip HTTP_EVADER_OK

This would then only run those tests that have HTTP_EVADER_OK in their requires and would output something like:
===> http-evader-447: FAILED requires HTTP_EVADER_OK, so considering OK
===> http-evader-448: OK - is requires HTTP_EVADER_OK needed?
..
PASSED:  1368
FAILED:  0
SKIPPED: 149
SKIPPED OK: 1
SKIPPED FAIL: 1

This would then make it clear which of the tests needs to be updated.

Related issues 2 (1 open — 1 closed)

Related to Suricata - Optimization #3172: Give only test count as outputClosedShivani BhardwajActions
Related to Suricata - Task #8532: suricata-verify: ensure CI covers all testsTriagedOISF DevActions

VJ Updated by Victor Julien almost 7 years ago Actions
Copy link
 #1

  • Target version set to QA

AH Updated by Andreas Herz almost 7 years ago Actions
Copy link
 #2

  • Assignee set to OISF Dev
  • Effort set to medium
  • Difficulty set to low

SB Updated by Shivani Bhardwaj almost 7 years ago Actions
Copy link
 #3

  • Status changed from New to Assigned
  • Assignee changed from OISF Dev to Shivani Bhardwaj

PM Updated by Peter Manev almost 7 years ago Actions
Copy link
 #4

off note - i just saw a small miss spell

 http-evader-448: OK - is requires ...

"is requires" should probably be "required".

(newbie :) ) Question - do we have a way to list what can be declared to be skipped ? Or how to list multiples tests sequences to be skipped?

JI Updated by Jason Ish almost 7 years ago Actions
Copy link
 #5

Related: You can tell suricata-verify what tests to run as well, for example:

.../run.py dns

will only run tests with "dns" in the name.

.../run.py dns http

will run tests with "dns" OR "http" in the name.

SB Updated by Shivani Bhardwaj almost 7 years ago Actions
Copy link
 #6

  • Status changed from Assigned to Feedback

SB Updated by Shivani Bhardwaj almost 7 years ago Actions
Copy link
 #7

  • Status changed from Feedback to Assigned

SB Updated by Shivani Bhardwaj over 6 years ago Actions
Copy link
 #8

SB Updated by Shivani Bhardwaj over 6 years ago Actions
Copy link
 #9

  • Status changed from Assigned to Feedback

SB Updated by Shivani Bhardwaj over 6 years ago Actions
Copy link
 #10

  • Status changed from Feedback to Assigned

SB Updated by Shivani Bhardwaj almost 6 years ago Actions
Copy link
 #11

  • Assignee changed from Shivani Bhardwaj to Community Ticket

SB Updated by Shivani Bhardwaj almost 6 years ago Actions
Copy link
 #12

  • Status changed from Assigned to New
  • Effort deleted (medium)
  • Difficulty deleted (low)
  • Label Outreachy, Python added

DM Updated by Dingani Muzimba over 5 years ago Actions
Copy link
 #13

  • Assignee changed from Community Ticket to Dingani Muzimba

JI Updated by Jason Ish almost 5 years ago Actions
Copy link
 #14

  • Assignee changed from Dingani Muzimba to Jason Ish
  • Label deleted (Outreachy)

VJ Updated by Victor Julien about 4 years ago Actions
Copy link
 #15

  • Status changed from New to Assigned
  • Assignee changed from Jason Ish to Shivani Bhardwaj

PA Updated by Philippe Antoine 2 months ago Actions
Copy link
 #16

  • Status changed from Assigned to Closed

We now have

--force Force running of skipped tests

SB Updated by Shivani Bhardwaj about 1 month ago Actions
Copy link
 #17

  • Status changed from Closed to Rejected
  • Assignee deleted (Shivani Bhardwaj)

Looks like this was closed as unneeded so changing the status to Rejected and removing myself from Assignee.

PA Updated by Philippe Antoine about 1 month ago Actions
Copy link
 #18

  • Status changed from Rejected to Closed

This was closed as implemented :-)

SB Updated by Shivani Bhardwaj about 1 month ago Actions
Copy link
 #19

@Philippe Antoine I'm failing to find this feature. :/

--force has been there for a long time and does not take args as is asked in the ticket.

PA Updated by Philippe Antoine about 1 month ago Actions
Copy link
 #20

I do not wee why we would need an argument (besides filtering based on the names of the tests)

PA Updated by Philippe Antoine about 1 month ago Actions
Copy link
 #21

  • Related to Task #8532: suricata-verify: ensure CI covers all tests added

PA Updated by Philippe Antoine about 1 month ago Actions
Copy link
 #22

One step after #8532 could be to run the skipped tests to see if they are indeed needed to be skipped

Actions
Copy link

Also available in: PDF Atom