Skip to content

Mobly exits with status code 0 after TestAbortAll #960

@puradox

Description

@puradox

If a test executed by test_runner raises signals.TestAbortAll before any failures or errors are received, the test will exit with status code 0 instead of what I expected to be 1. This is due to not executing ok = False

ok = True
for config in test_configs:
runner = TestRunner(
log_dir=config.log_path, testbed_name=config.testbed_name
)
with runner.mobly_logger(console_level=console_level):
runner.add_test_class(config, test_class, tests)
try:
runner.run()
ok = runner.results.is_all_pass and ok
except signals.TestAbortAll:
pass
except Exception:
logging.exception('Exception when executing %s.', config.testbed_name)
ok = False
if not ok:
sys.exit(1)

To make this more confusing, if a test failure or error is experienced before TestAbortAll is raised, the test will exit with status code 1.

I believe both of these behaviors oppose most users' expectations of the implication of aborting a test. I would expect the test to always return status code 1 whenever an abort occurs. WDYT about adding ok = False to the except signals.TestAbortAll statement?

At the very least, some doc comments above each signal type would be valuable for setting expectations.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions