Project

General

Profile

Actions
Copy link

Feature #3082

closed
VJ

suricata-verify: option to test 'skipped' tests

Feature #3082: suricata-verify: option to test 'skipped' tests

Added by Victor Julien almost 7 years ago. Updated 11 days ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
QA
Effort:
Difficulty:
Label:
Python

Description

Now that the number of tests is growing a new issue appears: which of the http evader tests are supposed to no longer be skipped after the last merges? We can try declaring HTTP_EVADER_OK and see which tests pass, but then we'd have to correlate that with the list of skipped tests if HTTP_EVADER_OK is not declared. Quite cumbersome.

So my suggestion is to add a way to check this. Run suricata-verify with an option to try tests with a skip condition anyway.

Something like:

run.py --test-skip HTTP_EVADER_OK

This would then only run those tests that have HTTP_EVADER_OK in their requires and would output something like:
===> http-evader-447: FAILED requires HTTP_EVADER_OK, so considering OK
===> http-evader-448: OK - is requires HTTP_EVADER_OK needed?
..
PASSED:  1368
FAILED:  0
SKIPPED: 149
SKIPPED OK: 1
SKIPPED FAIL: 1

This would then make it clear which of the tests needs to be updated.

Related issues 2 (1 open — 1 closed)

Related to Suricata - Optimization #3172: Give only test count as outputClosedShivani BhardwajActions
Related to Suricata - Task #8532: suricata-verify: ensure CI covers all testsTriagedOISF DevActions

VJ Updated by Victor Julien almost 7 years ago Actions
Copy link
 #1

  • Target version set to QA

AH Updated by Andreas Herz almost 7 years ago Actions
Copy link
 #2

  • Assignee set to OISF Dev
  • Effort set to medium
  • Difficulty set to low

SB Updated by Shivani Bhardwaj almost 7 years ago Actions
Copy link
 #3

  • Status changed from New to Assigned
  • Assignee changed from OISF Dev to Shivani Bhardwaj

PM Updated by Peter Manev almost 7 years ago Actions
Copy link
 #4

off note - i just saw a small miss spell

 http-evader-448: OK - is requires ...

"is requires" should probably be "required".

(newbie :) ) Question - do we have a way to list what can be declared to be skipped ? Or how to list multiples tests sequences to be skipped?

JI Updated by Jason Ish almost 7 years ago Actions
Copy link
 #5

Related: You can tell suricata-verify what tests to run as well, for example:

.../run.py dns

will only run tests with "dns" in the name.

.../run.py dns http

will run tests with "dns" OR "http" in the name.

SB Updated by Shivani Bhardwaj almost 7 years ago Actions
Copy link
 #6

  • Status changed from Assigned to Feedback

SB Updated by Shivani Bhardwaj almost 7 years ago Actions
Copy link
 #7

  • Status changed from Feedback to Assigned

SB Updated by Shivani Bhardwaj over 6 years ago Actions
Copy link
 #8

SB Updated by Shivani Bhardwaj over 6 years ago Actions
Copy link
 #9

  • Status changed from Assigned to Feedback

SB Updated by Shivani Bhardwaj over 6 years ago Actions
Copy link
 #10

  • Status changed from Feedback to Assigned

SB Updated by Shivani Bhardwaj over 5 years ago Actions
Copy link
 #11

  • Assignee changed from Shivani Bhardwaj to Community Ticket

SB Updated by Shivani Bhardwaj over 5 years ago Actions
Copy link
 #12

  • Status changed from Assigned to New
  • Effort deleted (medium)
  • Difficulty deleted (low)
  • Label Outreachy, Python added

DM Updated by Dingani Muzimba over 5 years ago Actions
Copy link
 #13

  • Assignee changed from Community Ticket to Dingani Muzimba

JI Updated by Jason Ish almost 5 years ago Actions
Copy link
 #14

  • Assignee changed from Dingani Muzimba to Jason Ish
  • Label deleted (Outreachy)

VJ Updated by Victor Julien about 4 years ago Actions
Copy link
 #15

  • Status changed from New to Assigned
  • Assignee changed from Jason Ish to Shivani Bhardwaj

PA Updated by Philippe Antoine about 1 month ago Actions
Copy link
 #16

  • Status changed from Assigned to Closed

We now have

--force Force running of skipped tests

SB Updated by Shivani Bhardwaj 11 days ago Actions
Copy link
 #17

  • Status changed from Closed to Rejected
  • Assignee deleted (Shivani Bhardwaj)

Looks like this was closed as unneeded so changing the status to Rejected and removing myself from Assignee.

PA Updated by Philippe Antoine 11 days ago Actions
Copy link
 #18

  • Status changed from Rejected to Closed

This was closed as implemented :-)

SB Updated by Shivani Bhardwaj 11 days ago Actions
Copy link
 #19

@Philippe Antoine I'm failing to find this feature. :/

--force has been there for a long time and does not take args as is asked in the ticket.

PA Updated by Philippe Antoine 11 days ago Actions
Copy link
 #20

I do not wee why we would need an argument (besides filtering based on the names of the tests)

PA Updated by Philippe Antoine 11 days ago Actions
Copy link
 #21

  • Related to Task #8532: suricata-verify: ensure CI covers all tests added

PA Updated by Philippe Antoine 11 days ago Actions
Copy link
 #22

One step after #8532 could be to run the skipped tests to see if they are indeed needed to be skipped

Actions
Copy link

Also available in: PDF Atom