Output seed information when `checking` fails during generation

[SevereOverfl0w]

If generation fails during the bindings phase of the checking macro any seed information is lost. This makes it difficult to reproduce failures.

(deftest fails
  (checking
    "a"
    [i gen/small-integer
     :let [x (/ 1 i)]]
    (is (pos? x))))

Produces this output:

ERROR in (fails) (Numbers.java:188)
Uncaught exception, not in assertion.
expected: nil
  actual: java.lang.ArithmeticException: Divide by zero
 at clojure.lang.Numbers.divide (Numbers.java:188)
    clojure.lang.Numbers.divide (Numbers.java:3877)
    com.gfredericks.test.chuck.clojure_test_test$fn__4276$fn__4277$fn__4284.invoke (clojure_test_test.cljc:55)
    clojure.test.check.rose_tree$fmap.invokeStatic (rose_tree.cljc:77)
    clojure.test.check.rose_tree$fmap.invoke (rose_tree.cljc:73)
    clojure.test.check.rose_tree$fmap$fn__618.invoke (rose_tree.cljc:77)
    clojure.core$map$fn__5884.invoke (core.clj:2759)
    clojure.lang.LazySeq.sval (LazySeq.java:42)
    clojure.lang.LazySeq.seq (LazySeq.java:51)
    clojure.lang.RT.seq (RT.java:535)
    clojure.core$seq__5419.invokeStatic (core.clj:139)
    clojure.core$map$fn__5884.invoke (core.clj:2750)
    clojure.lang.LazySeq.sval (LazySeq.java:42)
    clojure.lang.LazySeq.seq (LazySeq.java:51)
    clojure.lang.RT.seq (RT.java:535)
    clojure.core$seq__5419.invokeStatic (core.clj:139)
    clojure.core/seq (core.clj:139)
    clojure.test.check.rose_tree$permutations$iter__629__635$fn__636$iter__631__640$fn__641.invoke (rose_tree.cljc:102)
    clojure.lang.LazySeq.sval (LazySeq.java:42)
    clojure.lang.LazySeq.seq (LazySeq.java:51)
    clojure.lang.RT.seq (RT.java:535)
    clojure.core$seq__5419.invokeStatic (core.clj:139)
    clojure.core/seq (core.clj:139)
    clojure.test.check.rose_tree$permutations$iter__629__635$fn__636.invoke (rose_tree.cljc:102)
    clojure.lang.LazySeq.sval (LazySeq.java:42)
    clojure.lang.LazySeq.seq (LazySeq.java:51)
    clojure.lang.RT.seq (RT.java:535)
    clojure.core$seq__5419.invokeStatic (core.clj:139)
    clojure.core$map$fn__5884.invoke (core.clj:2750)
    clojure.lang.LazySeq.sval (LazySeq.java:42)
    clojure.lang.LazySeq.seq (LazySeq.java:51)
    clojure.lang.RT.seq (RT.java:535)
    clojure.core$seq__5419.invokeStatic (core.clj:139)
    clojure.core$map$fn__5884.invoke (core.clj:2750)
    clojure.lang.LazySeq.sval (LazySeq.java:42)
    clojure.lang.LazySeq.seq (LazySeq.java:51)
    clojure.lang.RT.seq (RT.java:535)
    clojure.core$seq__5419.invokeStatic (core.clj:139)
    clojure.core$empty_QMARK_.invokeStatic (core.clj:6195)
    clojure.core$empty_QMARK_.invoke (core.clj:6195)
    clojure.test.check$shrink_loop.invokeStatic (check.cljc:263)
    clojure.test.check$shrink_loop.invoke (check.cljc:242)
    clojure.test.check$failure.invokeStatic (check.cljc:314)
    clojure.test.check$failure.invoke (check.cljc:297)
    clojure.test.check$quick_check.invokeStatic (check.cljc:228)
    clojure.test.check$quick_check.doInvoke (check.cljc:59)
    clojure.lang.RestFn.invoke (RestFn.java:425)
    clojure.lang.AFn.applyToHelper (AFn.java:156)
    clojure.lang.RestFn.applyTo (RestFn.java:132)
    clojure.core$apply.invokeStatic (core.clj:671)
    clojure.core$apply.invoke (core.clj:662)
    com.gfredericks.test.chuck.clojure_test_test$fn__4276$fn__4277.invoke (clojure_test_test.cljc:52)
    com.gfredericks.test.chuck.clojure_test$_testing.invokeStatic (clojure_test.cljc:103)
    com.gfredericks.test.chuck.clojure_test$_testing.invoke (clojure_test.cljc:101)
    com.gfredericks.test.chuck.clojure_test_test$fn__4276.invokeStatic (clojure_test_test.cljc:52)
    com.gfredericks.test.chuck.clojure_test_test/fn (clojure_test_test.cljc:51)
    clojure.test$test_var$fn__9761.invoke (test.clj:717)
    clojure.test$test_var.invokeStatic (test.clj:717)
    clojure.test$test_var.invoke (test.clj:708)
    clojure.test$test_vars$fn__9787$fn__9792.invoke (test.clj:735)
    clojure.test$default_fixture.invokeStatic (test.clj:687)
    clojure.test$default_fixture.invoke (test.clj:683)
    clojure.test$test_vars$fn__9787.invoke (test.clj:735)
    clojure.test$default_fixture.invokeStatic (test.clj:687)
    clojure.test$default_fixture.invoke (test.clj:683)
    clojure.test$test_vars.invokeStatic (test.clj:731)
    clojure.test$test_all_vars.invokeStatic (test.clj:737)
    clojure.test$test_ns.invokeStatic (test.clj:758)
    clojure.test$test_ns.invoke (test.clj:743)
    user$eval227$fn__350.invoke (form-init6834647596943620637.clj:1)
    clojure.lang.AFn.applyToHelper (AFn.java:156)
    clojure.lang.AFn.applyTo (AFn.java:144)
    clojure.core$apply.invokeStatic (core.clj:669)
    clojure.core$apply.invoke (core.clj:662)
    leiningen.core.injected$compose_hooks$fn__154.doInvoke (form-init6834647596943620637.clj:1)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invokeStatic (core.clj:667)
    clojure.core$apply.invoke (core.clj:662)
    leiningen.core.injected$run_hooks.invokeStatic (form-init6834647596943620637.clj:1)
    leiningen.core.injected$run_hooks.invoke (form-init6834647596943620637.clj:1)
    leiningen.core.injected$prepare_for_hooks$fn__159$fn__160.doInvoke (form-init6834647596943620637.clj:1)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.lang.AFunction$1.doInvoke (AFunction.java:31)
    clojure.lang.RestFn.invoke (RestFn.java:408)
    clojure.core$map$fn__5884.invoke (core.clj:2759)
    clojure.lang.LazySeq.sval (LazySeq.java:42)
    clojure.lang.LazySeq.seq (LazySeq.java:51)
    clojure.lang.Cons.next (Cons.java:39)
    clojure.lang.RT.boundedLength (RT.java:1793)
    clojure.lang.RestFn.applyTo (RestFn.java:130)
    clojure.core$apply.invokeStatic (core.clj:669)
    clojure.test$run_tests.invokeStatic (test.clj:768)
    clojure.test$run_tests.doInvoke (test.clj:768)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invokeStatic (core.clj:667)
    clojure.core$apply.invoke (core.clj:662)
    user$eval227$fn__360$fn__413.invoke (form-init6834647596943620637.clj:1)
    user$eval227$fn__360$fn__361.invoke (form-init6834647596943620637.clj:1)
    user$eval227$fn__360.invoke (form-init6834647596943620637.clj:1)
    user$eval227.invokeStatic (form-init6834647596943620637.clj:1)
    user$eval227.invoke (form-init6834647596943620637.clj:1)
    clojure.lang.Compiler.eval (Compiler.java:7181)
    clojure.lang.Compiler.eval (Compiler.java:7171)
    clojure.lang.Compiler.load (Compiler.java:7640)
    clojure.lang.Compiler.loadFile (Compiler.java:7578)
    clojure.main$load_script.invokeStatic (main.clj:475)
    clojure.main$init_opt.invokeStatic (main.clj:477)
    clojure.main$init_opt.invoke (main.clj:477)
    clojure.main$initialize.invokeStatic (main.clj:508)
    clojure.main$null_opt.invokeStatic (main.clj:542)
    clojure.main$null_opt.invoke (main.clj:539)
    clojure.main$main.invokeStatic (main.clj:664)
    clojure.main$main.doInvoke (main.clj:616)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.lang.Var.applyTo (Var.java:705)
    clojure.main.main (main.java:40)

Ran 5 tests containing 20 assertions.
0 failures, 1 errors.
Tests failed.

[SevereOverfl0w]

After poking at this a little bit, it seems that we'd need upstream changes in test.check to make this work. When doing a call-gen in quick-check the exception would need to be caught and some handling put in place. Calling :reporter-fn with any relevant information before rethrowing the exception seems like the minimal set of changes to me.

[gfredericks]

Okay; we can leave this open and you or somebody else could pursue those changes in test.check.

[frenchy64]

@gfredericks I took a shot at it. #80

What do you think about the test.check changes? Worth proposing on TCHECK-159?

https://github.com/frenchy64/test.check/pull/1/files

[gfredericks]

I'm not maintaining test.check anymore, so my opinion doesn't matter too much.

[frenchy64]

Firstly, thanks @gfredericks for all the years of maintainership. In my book, test.check under your direction has been the most successful Clojure library of them all. All that to say, I really appreciate your work and congratulations on an amazing tenure.

Secondly, my hit rate on getting patches merged from JIRA is probably about 5%. Would you consider a patch to test.chuck with this definition of quick-check, and then using it in checking?

[gfredericks]

Thanks for the kind words!

Copypasting the definition of quickcheck into this library sounds like it could gen gnarly in the future if implementation details in test.check change.

Is there no way to do this in test.chuck, perhaps by wrapping generators with a fmap that has a try/catch, and then wrapping the checking expression similarly? That'd amount to turning a generator-exception into a test failure, which is a bit weird, but would at least result in the seed being output.

[frenchy64]

@gfredericks great idea, I will try it.