How to run this project without using slurm？

[liujian123223]

Hello！
First of all, thank you for your excellent work, which has greatly inspired me. You provided the sbatch file for running the project using slurm in the readme document. But I only have one computer equipped with the Ubuntu system. May I ask how to run your code without using Slurm?

Looking forward to your reply.

[jaggbow]

Hi!

You can just use the command in the sbatch file (the one that starts with python) and run that on your machine. I recently updated the paths in the config and they should work on your machine. Let me know if that works!

[liujian123223]

Thank you for your reply. I followed your instructions and ran the program using the following method~/Project/CrossViVit $Python main. py experience=cross_vit. But there will be an error message:

omegaconf.errors.InterpolationResolutionError: KeyError raised while resolving interpolation: "Environment variable 'SLURM_JOB_ID' not found"
full_key: hydra.run.dir
object_type=dict

So I changed the section in the. yaml file where "${oc. env: SLIRM_JOB-ID}" appeared to "1". For example:

dir: ${paths.log_dir}/${task_name}/multiruns/${oc.env:SLURM_JOB_ID}
dir: ${paths.log_dir}/${task_name}/multiruns/1

Then the program can continue to run until:

wandb: (1) Create a W&B account
wandb: (2) Use an existing W&B account
wandb: (3) Don't visualize my results

If I choose (3), the code will run to the following state and will not change for a long time, and I find that the GPU is almost unused.If I choose to log in to my Wandb account, I will receive an error message
Wandb: Error Error while calling W&B API: project not found (<Response [404]>)

I still haven't solved my problem. I hope to be able to run your program locally (without Slurm environment), and I hope you can help me analyze where the problem lies


[liujian123223]

Hi, I have found that the long running time mentioned before is due to the process of importing the SunLake dataset from Deeplake, which takes a very long time (more than 2 hours). Do you know of any solutions to this problem?
ds = deeplake.load('hub://crossvivit/SunLake')

[jaggbow]

Thank you for your reply. I followed your instructions and ran the program using the following method~/Project/CrossViVit $Python main. py experience=cross_vit. But there will be an error message:

omegaconf.errors.InterpolationResolutionError: KeyError raised while resolving interpolation: "Environment variable 'SLURM_JOB_ID' not found"
full_key: hydra.run.dir
object_type=dict

So I changed the section in the. yaml file where "${oc. env: SLIRM_JOB-ID}" appeared to "1". For example:

dir: ${paths.log_dir}/${task_name}/multiruns/${oc.env:SLURM_JOB_ID}
dir: ${paths.log_dir}/${task_name}/multiruns/1

Then the program can continue to run until:

wandb: (1) Create a W&B account
wandb: (2) Use an existing W&B account
wandb: (3) Don't visualize my results

If I choose (3), the code will run to the following state and will not change for a long time, and I find that the GPU is almost unused.If I choose to log in to my Wandb account, I will receive an error message Wandb: Error Error while calling W&B API: project not found (<Response [404]>)

I still haven't solved my problem. I hope to be able to run your program locally (without Slurm environment), and I hope you can help me analyze where the problem lies

For the wandb problem, make sure that the wandb project you create in your account matches the one in the config file: configs/logger/wandb.yaml

[jaggbow]

Hi, I have found that the long running time mentioned before is due to the process of importing the SunLake dataset from Deeplake, which takes a very long time (more than 2 hours). Do you know of any solutions to this problem? ds = deeplake.load('hub://crossvivit/SunLake')

You can download the deeplake data locally which should speed up things. We're also working on moving away from deeplake and providing the dataset in another format like zarr or hdf5 but it will require changing the dataloader accordingly.

[liujian123223]

Hi, I have found that the long running time mentioned before is due to the process of importing the SunLake dataset from Deeplake, which takes a very long time (more than 2 hours). Do you know of any solutions to this problem? ds = deeplake.load('hub://crossvivit/SunLake')

You can download the deeplake data locally which should speed up things. We're also working on moving away from deeplake and providing the dataset in another format like zarr or hdf5 but it will require changing the dataloader accordingly.

Can you tell me where I can download deeplake data locally？