Badly-behaved tokio task can wedge the runtime #9619
Description
The root cause of #9569 was twofold:
- A tokio task in trust quorum could, under unusual-but-expected conditions, go into an infinite loop due to not handling
read()returning 0. This is fixed by TQ: Fix potential infinite read loop and tokio hang #9612. - This single task's behavior caused the entire tokio runtime to hang. That's the topic of this issue.
The relevant tokio issues are tokio-rs/tokio#4730, the title of which exactly matches the behavior we've observed here but is closed, and tokio-rs/tokio#6315, an open issue about better tolerating badly-behaved tasks. tokio-rs/tokio#6315 (comment) describes three ways that long polls (in our case, an infinitely long poll) can degrade performance (in our case, hang the runtime entirely), and notes that that issue is about the third way:
- There's a single blocking future on the thread that is currently responsible for receiving incoming IO/timer events.
The description of how to trigger this is:
- All threads are idle. Thread A is waiting for IO/timer events.
- Thread A receives an IO or timer event and wakes up to handle it.
- A single task is woken up by the IO/timer event, and thread A polls it.
- During the call to poll, a new IO (or timer) event becomes available.
- Thread A does not see the IO event since it's polling a future. The other worker threads don't see it either, since none of them are currently registered to receive IO events.
This was extremely helpful in narrowing down what was happening in #9569. That same comment also notes that it is possible to unwedge the runtime in this case, if a separate thread is spawned that periodically injects a do-nothing task that forces the runtime to wake up:
None of this happens if there are other active workers, because they will take over IO events when they next become idle. This is why you can make your own monitor thread by regularly spawning tasks on the runtime.
I'm strongly tempted to suggest we add this pattern to all of our tokio-based binaries, although it has upsides and downsides. We have to choose an interval on which to have this monitor thread inject a task. Any hang will recover at the next task injection point, so if we make this too long we can still have large stalls, and if we make it too short we've got a bunch of otherwise-worthless CPU work. (That's technically true regardless of how long or short the interval is - this thread does nothing useful except "try to make sure tokio isn't wedged".)
I'll post a comment below with some notes about the particular debugging we did to confirm this is the situation we were in and some signs to look for if this comes up again.