suricata-verify: option to test 'skipped' tests
Now that the number of tests is growing a new issue appears: which of the http evader tests are supposed to no longer be skipped after the last merges? We can try declaring HTTP_EVADER_OK and see which tests pass, but then we'd have to correlate that with the list of skipped tests if HTTP_EVADER_OK is not declared. Quite cumbersome.
So my suggestion is to add a way to check this. Run suricata-verify with an option to try tests with a skip condition anyway.
run.py --test-skip HTTP_EVADER_OK
This would then only run those tests that have HTTP_EVADER_OK in their requires and would output something like:
===> http-evader-447: FAILED requires HTTP_EVADER_OK, so considering OK ===> http-evader-448: OK - is requires HTTP_EVADER_OK needed? .. PASSED: 1368 FAILED: 0 SKIPPED: 149 SKIPPED OK: 1 SKIPPED FAIL: 1
This would then make it clear which of the tests needs to be updated.
Updated by Peter Manev 8 days ago
off note - i just saw a small miss spell
http-evader-448: OK - is requires ...
"is requires" should probably be "required".
(newbie :) ) Question - do we have a way to list what can be declared to be skipped ? Or how to list multiples tests sequences to be skipped?