[JENKINS-25519] If unable to read exitStatus file then wait briefly and retry one more time to allow write to complete

[svanoort]

Proposed solution to error like this, encountered when statusCode file exists but is empty, potentially when created but not written to yet, or not written fully:

java.lang.NumberFormatException: For input string: ""
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:592)
at java.lang.Integer.parseInt(Integer.java:615)
at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.exitStatus(FileMonitoringTask.java:168)
Caused: java.io.IOException: corrupted content in $SOMEPLACE
at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.exitStatus(FileMonitoringTask.java:170)
at org.jenkinsci.plugins.durabletask.BourneShellScript$ShellController.exitStatus(BourneShellScript.java:211)

Variant of JENKINS-25519

This may not be the best solution to the problem in my opinion but should generally solve it.

Other options:

• Return null until the file parses cleanly (normal timeout will apply once the launching script dies and log file no longer is touched).
• Write and look for a delimiter on the exitStatus file to indicate that the file is fully written (avoids theoretical issues with only '1' being written when status code might actually be '102' or something).

[jglick]

I think it is safe enough, if the problem was in fact diagnosed correctly. I swear there was some earlier attempt to fix this, but I cannot find it. JENKINS-25519 describes a similar-sounding case but note that the exception is repeatedly thrown—meaning that the file was created but stayed empty, for some reason. So this should perhaps be considered only a hotfix:

• root cause not obvious (is it really just a simple race condition?)
• caller should perhaps treat this as fatal

[jglick]

Well this alone would fix the issue, right?

[svanoort]

Theoretically, yes, if there's no bogus whitespace or other stuff in there.

[jglick]

Ah, maybe I was thinking of #37—similar problem (at least on the face of it), different file.

If the problem is indeed a simple race condition, tricks like this are not the best fix. Rather, the wrapper script should be changed from

…; echo $? > …/exit

to

…; echo $? > …/exit.tmp; mv …/exit.tmp …/exit

which ought to be atomic.

[svanoort]

@jglick Has requested a test-comment, please ignore this. @cloudbees/team-arc

[svanoort]

Closing in favor of #66