git/linux-cryptodev-2.6
1
0
Fork 0
mirror of https://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git synced 2026-04-23 05:56:14 -04:00
Code Issues Packages Projects Releases Wiki Activity

kunit: tool: print summary of failed tests if a few failed out of a lot

Browse Source 
E.g. all the hw_breakpoint tests are failing right now.
So if I run `kunit.py run --altests --arch=x86_64`, then I see
> Testing complete. Ran 408 tests: passed: 392, failed: 9, skipped: 7

Seeing which 9 tests failed out of the hundreds is annoying.
If my terminal doesn't have scrollback support, I have to resort to
looking at `.kunit/test.log` for the `not ok` lines.

Teach kunit.py to print a summarized list of failures if the # of tests
reachs an arbitrary threshold (>=100 tests).

To try and keep the output from being too long/noisy, this new logic
a) just reports "parent_test failed" if every child test failed
b) won't print anything if there are >10 failures (also arbitrary).

With this patch, we get an extra line of output showing:
> Testing complete. Ran 408 tests: passed: 392, failed: 9, skipped: 7
> Failures: hw_breakpoint

This also works with parameterized tests, e.g. if I add a fake failure
> Failures: kcsan.test_atomic_builtins_missing_barrier.threads=6

Note: we didn't have enough tests for this to be a problem before.
But with commit 980ac3ad05 ("kunit: tool: rename all_test_uml.config,
use it for --alltests"), --alltests works and thus running >100 tests
will probably become more common.

Signed-off-by: Daniel Latypov <dlatypov@google.com>
Reviewed-by: David Gow <davidgow@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
This commit is contained in:
Daniel Latypov
2022-10-28 14:02:56 -07:00
committed by Shuah Khan
parent 3ffdcf7e3b
commit f19dd011d8
2 changed files with 69 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
tools/testing/kunit/kunit_tool_test.py
View File

@@ -312,6 +312,28 @@ class KUnitParserTest(unittest.TestCase):
result.status)
self.assertEqual('kunit-resource-test', result.subtests[0].name)
def test_summarize_failures(self):
output = """
KTAP version 1
1..2
# Subtest: all_failed_suite
1..2
not ok 1 - test1
not ok 2 - test2
not ok 1 - all_failed_suite
# Subtest: some_failed_suite
1..2
ok 1 - test1
not ok 2 - test2
not ok 1 - some_failed_suite
"""
result = kunit_parser.parse_run_tests(output.splitlines())
self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status)
self.assertEqual(kunit_parser._summarize_failed_tests(result),
'Failures: all_failed_suite, some_failed_suite.test2')
def line_stream_from_strs(strs: Iterable[str]) -> kunit_parser.LineStream:
return kunit_parser.LineStream(enumerate(strs, start=1))