test_fisher_vec_prod occasionally fails on Jenkins #407
Description
Every so often, tests/test_eigendistortion.py::TestAutodiffFunctions::test_fisher_vec_prod will occasionally fail on Jenkins as part of our GPU tests. I have not been able to reproduce it locally and it doesn't happen consistently, so I'm not quite sure how to track it down. I had originally thought it was the result of getting different nodes, which had slightly different CUDA drivers, but that doesn't appear to be the case: I'm uploading the logs of an example successful and failed run, both from #392, one after the other. Before the tests run (and after setting up the environment), you can see the output of nvidia-smi, and it looks the same on both. So not sure how to proceed here.