Skip to content

RuntimeError: CUDA out of memory #11

@hitbuyi

Description

@hitbuyi
(pt110) hitbuyi@hitbuyi-Dell-G15-5511:~/AD_Projects/Pytorch_Project/LMSCNet$ python LMSCNet/train.py --cfg SSC_configs/examples/LMSCNet.yaml --dset_root ./dataset_kitti
2024-07-23 21:04:29,065 -- ============ Training routine: "SSC_configs/examples/LMSCNet.yaml" ============

2024-07-23 21:04:29,861 -- => Loading network architecture...
2024-07-23 21:04:29,926 -- => Loading optimizer...
2024-07-23 21:04:29,927 -- => No checkpoint. Initializing model from scratch
2024-07-23 21:04:35,266 -- => =============== Epoch [1/80] ===============
2024-07-23 21:04:35,266 -- => Reminder - Output of routine on ../SSC_out/LMSCNet_SemanticKITTI_0723_203809
/home/hitbuyi/.conda/envs/pt110/lib/python3.8/site-packages/torch/optim/lr_scheduler.py:247: UserWarning: To get the last learning rate computed by the scheduler, please use `get_last_lr()`.
  warnings.warn("To get the last learning rate computed by the scheduler, "
2024-07-23 21:04:35,266 -- => Learning rate: 0.001
Traceback (most recent call last):
  File "LMSCNet/train.py", line 333, in <module>
    main()
  File "LMSCNet/train.py", line 319, in main
    best_record = train(model, optimizer, scheduler, dataset, _cfg, epoch, logger, tbwriter)
  File "LMSCNet/train.py", line 107, in train
    loss['total'].backward()
  File "/home/hitbuyi/.conda/envs/pt110/lib/python3.8/site-packages/torch/_tensor.py", line 307, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/hitbuyi/.conda/envs/pt110/lib/python3.8/site-packages/torch/autograd/__init__.py", line 154, in backward
    Variable._execution_engine.run_backward(
RuntimeError: CUDA out of memory. Tried to allocate 1.69 GiB (GPU 0; 5.79 GiB total capacity; 1.88 GiB already allocated; 1.52 GiB free; 1.93 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

How much the CUDA memory is required to run this project?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions