-
Notifications
You must be signed in to change notification settings - Fork 205
Description
If a test executed by test_runner raises signals.TestAbortAll before any failures or errors are received, the test will exit with status code 0 instead of what I expected to be 1. This is due to not executing ok = False
Lines 71 to 87 in 991ecb2
| ok = True | |
| for config in test_configs: | |
| runner = TestRunner( | |
| log_dir=config.log_path, testbed_name=config.testbed_name | |
| ) | |
| with runner.mobly_logger(console_level=console_level): | |
| runner.add_test_class(config, test_class, tests) | |
| try: | |
| runner.run() | |
| ok = runner.results.is_all_pass and ok | |
| except signals.TestAbortAll: | |
| pass | |
| except Exception: | |
| logging.exception('Exception when executing %s.', config.testbed_name) | |
| ok = False | |
| if not ok: | |
| sys.exit(1) |
To make this more confusing, if a test failure or error is experienced before TestAbortAll is raised, the test will exit with status code 1.
I believe both of these behaviors oppose most users' expectations of the implication of aborting a test. I would expect the test to always return status code 1 whenever an abort occurs. WDYT about adding ok = False to the except signals.TestAbortAll statement?
At the very least, some doc comments above each signal type would be valuable for setting expectations.