NUM_GPUS

[2561681244]

hello! It is OK when NUM_GPUS is set to 1, but we encountered a problem as described in the picture when NUM_GPUS is set to 4(larger than 1).I don't know why.

[OrangeSodahub]

@2561681244 Hi, did you successfully get the ultimate results using 1 GPU for training?

In my case, I always meet the issue of torch out of memory, depite I reduce the batch size to extremely low values. I was thinking if I should not use only 1 GPU since there are multiple processes running with Ray.

[2561681244]

@2561681244 Hi, did you successfully get the ultimate results using 1 GPU for training?

In my case, I always meet the issue of torch out of memory, depite I reduce the batch size to extremely low values. I was thinking if I should not use only 1 GPU since there are multiple processes running with Ray.

I haven't got the ultimate results using 1 GPU for training,because only 4 A100(40G) were available.H100(80G) is difficult to book.I will continue to reproduce when I make an appointment for H100.

[OrangeSodahub]

@2561681244 I am using L20 40GB. I reduced the batch size to make the codebase running on 40GB card.

Did you have OOM issue when using 1 GPU?

[2561681244]

@2561681244 I am using L20 40GB. I reduced the batch size to make the codebase running on 40GB card.

Did you have OOM issue when using 1 GPU?

Yeah，I encountered the same problem with you.So I need to book H100.

[OrangeSodahub]

@2561681244 But when I was using 1 GPU (A800 80GB), OOM issue still exist.

[2561681244]

@2561681244 But when I was using 1 GPU (A800 80GB), OOM issue still exist.

Oh my god! I don't know the reason either.You can look at other issues, they may encounter the same problems as you.

[OrangeSodahub]

@2561681244 The OOM issue occurred after running for ~8 hours when I use 1 GPU with 80GB. So what I am asking is if you have the same problem with me. You just mentioned it is OK to use 1 GPU but did you finish the entire training sucessfully or not?

[2561681244]

@2561681244 The OOM issue occurred after running for ~8 hours when I use 1 GPU with 80GB. So what I am asking is if you have the same problem with me. You just mentioned it is OK to use 1 GPU but did you finish the entire training sucessfully or not?

I haven't finished the whole training.In fact, I encountered the OOM problem after running for 2 minutes.