`skvideo.io.FFmpegWriter.writeFrame` doesn't write frames deterministically?

[t-sasatani]

There appears to be an issue with skvideo.io's writeFrame function, where it sometimes fails to write frames without generating any exceptions. I found this while investigating why the maximum number of reconstructed_frame_index doesn't match the number of frames in the exported .avi file. We need to strictly match the data stream buffer index with the output video frame index to synchronize video timestamps and develop video repair functions; this issue is critical for these usages.

Summary

• The number of non-excepted skvideo.io.FFmpegWriter.writeFrame calls doesn't match the number of frames in the exported video
• There is a reshape and casting for the input array before calling writeFrame, so it's unlikely that the input data is broken.

Status

Failing tests for this issue are in:

• Bugfix: mio stream reconstructed frame index error #137
• https://github.com/Aharoni-Lab/mio/blob/bugfix_reconstruct_frame_count/tests/test_stream_daq.py#L342C1-L377C1

This can also run by:

pdm run pytest tests/test_stream_daq.py::test_writer_returns_match_avi_frame_count

which currently results in:

FAILED tests/test_stream_daq.py::test_writer_returns_match_avi_frame_count - AssertionError: write_frame True returns (116) != AVI frames (94)

[t-sasatani]

Worst worst case, we could count frames of the video export every loop.

[t-sasatani]

I thought specifying the fps in input_dict (bottom) would solve this issue entirely. However, I noticed that:

• The silent frame drops still happen while working on the behavioral camera stuff in Feature webcam #144 (windows with MJPEG or h264 export).
• This doesn't happen with cv2-based video writing.

So I added an optional cv2 backend to the VideoWriter to avoid this issue, though that is not optimal.

The skvideo.io config that I thought would solve thi,s but didn't entirely:

    input_dict = {"-framerate": str(fps)}
    output_dict = {
        [other stuff]
        "-vsync": "0", # probably not necessary but for safety
    }