[BUG]: cufile test_set_stats_level test fails in CI

[mdboom]

Is this a duplicate?

• I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct

Type of Bug

Something else

Component

cuda.pathfinder

Describe the bug

This test test_set_stats_level fails when run in CI. @chloechia4 and I have tried to get more information out by dumping the cufile.log after the run, but it appears to be empty. In order to not hold up #1060 any longer, I have marked this test as skip and am creating this issue to come back to it later.

=================================== FAILURES ===================================
_____________________________ test_set_stats_level _____________________________

    @pytest.mark.skipif(
        cufileVersionLessThan(1140), reason="cuFile parameter APIs require cuFile library version 13.0 or later"
    )
    def test_set_stats_level():
        """Test cuFile statistics level configuration."""
        # Initialize CUDA
        (err,) = cuda.cuInit(0)
        assert err == cuda.CUresult.CUDA_SUCCESS
    
        err, device = cuda.cuDeviceGet(0)
        assert err == cuda.CUresult.CUDA_SUCCESS
    
        err, ctx = cuda.cuDevicePrimaryCtxRetain(device)
        assert err == cuda.CUresult.CUDA_SUCCESS
        (err,) = cuda.cuCtxSetCurrent(ctx)
        assert err == cuda.CUresult.CUDA_SUCCESS
    
        # Open cuFile driver
>       cufile.driver_open()

tests/test_cufile.py:1926: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
cuda/bindings/cufile.pyx:2571: in cuda.bindings.cufile.driver_open
    cpdef driver_open():
cuda/bindings/cufile.pyx:2578: in cuda.bindings.cufile.driver_open
    check_status(__status__)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   raise cuFileError(status.err, status.cu_err)
E   cuda.bindings.cufile.cuFileError: DRIVER_NOT_INITIALIZED (5001): nvidia-fs driver is not loaded. Set allow_compat_mode to true in cufile.json file to enable compatible mode; CUDA status: CUDA_SUCCESS (0)

cuda/bindings/cufile.pyx:2467: cuFileError

How to Reproduce

Run the test_cufile.py test suite.

Expected behavior

Tests pass.

Operating System

No response

nvidia-smi output

No response

[chloechia4]

One note, test_set_stats_level in this snippet should be cufile version 11.50 (instead of 11.40). This is a special API added for 13.0. Changes are reflected in the original PR. But the tests still fail regardless

[leofang]

The bug is not in the tests Chloe added in #1060 (after her most recent fix), but in the 3 test_set_* tests that @sourabgupta3 added when the cuFILE bindings were introduced. All tests have side effects that propagate to other tests, causing irrelevant tests to fail. I have fixed it in Chloe's PR (e1eabf8).