torch/cuda/whatever keeps freezing

[sharpobject]

Seriously it will just freeze sometimes. idk if this is a driver issue or something. I thought it was a luajit bug, but it still happens sometimes with the jit disabled, so that's probably not it. Solution: Make some OTHER process watch the training process and restart it if it isn't working.

[sharpobject]

After messing with this more I no longer think it's happening in torch code. But I don't know how it's happening. I should maybe try using a debugger.