On failure, the test suite just prints info to stdout. It does not actually fail the process with an exit code.
So automated testing is impossible.
Usually, test suited declared as type: exitcode-stdio-1.0 are supposed to signal failure with an exit code.