Workaround for solver returning NaN

[emmetfrancis]

Detect when dolfin.assemble_mixed returns nan and recompute (in smart.solvers)

Right now, this is only implemented in one place for the residual calculation that seemed to usually cause the issue, but there may be other times that dolfin.assemble_mixed could return nan as well.

[jorgensd]

I would be good to figure out the root cause of this. Do you have a minimal example?

[emmetfrancis]

Agreed, I do not have a minimal working example yet, it is not clear why this returns NaN sometimes, but I was able to narrow it down to these lines. The strangest part is that this seems to occur non-deterministically, Henrik also observed this issue.

I can come back to this in the next couple days to hopefully put together a MWE, it's a confusing one... in the meantime feel free to let me know if you have additional thoughts. But we can wait to merge this I think.

[jorgensd]

Agreed, I do not have a minimal working example yet, it is not clear why this returns NaN sometimes, but I was able to narrow it down to these lines. The strangest part is that this seems to occur non-deterministically, Henrik also observed this issue.

I think there is a map from one mesh to another that is not created in time (where it should). I’ve seen this before and ive fixed those instances in the latest release of dolfin. Clearly there must be some cases ive missed.

[finsberg]

I agree with Jørgen we should try to find the root cause of this. My main concern with this fix is that there is a risk that the program enters an infinite loop that it cannot escape. If we go for this solution, there should be a guard against that, e.g that it tries N number of times, or for N seconds before it breaks out of the loop.

[emmetfrancis]

Added protection against infinite loop.