Issue03 test and coverage thresholds

[jhenligne]

Here are explanations about thresholds implementation and CircleCI integration.

Test coverage threshold

Use of hpc-threshold

hpc-threshold utility implements desired behavior.
Setup is straightforward:

• utility install:
  $ stack install hpc-threshold
• creation of a .hpc-threshold configuration file (threshold already set to 80%)
• build of a coverage report:
  $ stack test :confcrypt-detailed-tests --coverage
• generation of a test report analyzed by hpc-threshold:
  stack hpc report --all 2>&1 | hpc-threshold

Results as of February 5, 2019

Report done on last confcrypt commit with one of Add test (Issue #10) uncommented:

Code coverage threshold check: FAIL
· Expressions used: 49.0% (< 80.0%)
· Boolean coverage: 25.0% (< 80.0%)
· Alternatives used: 54.0% (< 80.0%)
· Local declarations used: 51.0% (< 80.0%)
· Top-level declarations used: 29.0% (< 80.0%)

Failed tests threshold

Implementation in ConsoleReporter.hs

The tasty-ant-xml package gave me a good basis to work out this solution.
Right now the Tasty package API only give access to individual tests while they are run, through the foldTestTree and trivialFold "callback" mechanism.
To ease analysis or format conversions, it would be useful to extend this behavior and provide access to an intermediate representation of results, once tests are completed. It may be that I will submit a PR to Tasty about this feature.

Tests reorganization

We now have 2 test suites.
The original one has been renamed from confcrypt-test to confcrypt-detailed-tests. In the same manner, Tests.hs module was renamed to DetailedTests.hs.
A new one, confcrypt-threshold-tests, is calling a corresponding ThresholdTests.hs module.

Usage

To run the existing test suite:
$ stack test :confcrypt-detailed-tests

To run the new one:

• Default 80% threshold: $ stack test :confcrypt-threshold-tests
• Custom threshold: $ stack test :confcrypt-threshold-tests --ta "--threshold 90"

This new test suite returns an error exit code if the threshold is not reached.

The general command stack test should not be used: the same set of tests would be run twice, differing only in their output.

Results as of February 5, 2019

Tests run on last confcrypt commit with one of Add test (Issue #10) uncommented and threshold set to 90%:

$ stack test :confcrypt-threshold-tests --ta "--threshold 90"
confcrypt-0.1.0.3: test (suite: confcrypt-threshold-tests, args: --threshold 90)


Number of tests: 47.0, Threshold: 90.0% => 43 tests
Failures: 1 (2.13%), Errors: 0 (0.0%), Successes: 46 (97.87%)

confcrypt-0.1.0.3: Test suite confcrypt-threshold-tests passed

CircleCI integration

Context

Not having access to CircleCI's environment, what follows are preparatory steps based on some observations and assumptions.

The actual config.yml configuration is a mix of CircleCI's commands and bash scripts calls.
Not knowing what limitations are imposed on those scripts I updated the config.yml file with both scenarios in mind: 1) using CircleCI' way to run commands, or 2) delegating this task to plain old scripts.

Probing

Two scripts are prepared as an alternative to the test-job job and are to be tested on site (both are running fine on my laptop):

• threshold-test.sh runs threshold tests and prepare a hpc report,
• coverage.sh outputs the hpc-threshold report in a hpc-threshold.log file.

Each script returns an error exit code if the test or the coverage threshold is not reached.

Usage

As explained previously, a test suite has to be chosen; running both would be redundant.
It means one command has to be commented while the other is uncommented.

If scripts are running fine in CircleCI's environment, test-job job could be rewritten by following the pattern of threshold-test.sh and the choice will be between test-job and threshold-test-job within the workflows section.

Otherwise, the switch will be done in the run command of the test-job job and a way to produce the hpc-threshold report will have to be found "à la CircleCI".

[jhenligne]

hpc-threshold can't be installed with ghc 8.4.3, it is failing with the pcre-light-0.4.0.4 package.

I will try to install CircleCI with Docker locally to debug this install setup.

Meanwhile if you know how to fix this problem you are welcome.

[jhenligne]

Integration done, failing test because code coverage under threshold:

#!/bin/bash -eo pipefail
stack hpc report --all 2>&1 | hpc-threshold

Code coverage threshold check: FAIL
· Expressions used: 46.0% (< 80.0%)
· Boolean coverage: 16.0% (< 80.0%)
· Alternatives used: 52.0% (< 80.0%)
· Local declarations used: 50.0% (< 80.0%)
· Top-level declarations used: 28.0% (< 80.0%)

Exited with code 1

@ChrisCoffey This PR is ready for review.