minimum requirements for a gpu #1
Description
Hello.
Thanks for your work.
What are the minimum requirements for a gpu to run your solution on test data (deepspeed_xl_runormas_v14_130k_finetune.sh) ?
I try to use this sh file with some changes in paths:
NUM_GPUS_PER_WORKER=0
gpt_options=" \
--train-data-path /home/mdomrachev/aspect-extractor/entities_normalizer/RuNormAS-solution/data/public_test_v13/files.list \
--max-files-per-process 10 \
--logging-dir=/mnt/de420497-d399-4936-af66-6bddbd9bfd33/Data/aspect_miner/demo/ \
--load /mnt/de420497-d399-4936-af66-6bddbd9bfd33/Data/aspect_miner/mp_rank_00_model_states.pt \
--save /mnt/de420497-d399-4936-af66-6bddbd9bfd33/Data/aspect_miner/demo/ \
--tokenizer-path sberbank-ai/rugpt3xl \
--no-load-optim \
--finetune \
--cache-prefix p16 \
--save-interval 5000 \
--log-interval 100 \
--model-parallel-size 1 \
--num-layers 24 \
--hidden-size 2048 \
--num-attention-heads 16 \
--batch-size 1 \
--seq-length 2048 \
--max-position-embeddings 2048 \
--train-iters 20000 \
--distributed-backend nccl \
--lr 0.000015 \
--warmup 0.0 \
--lr-decay-style constant \
--weight-decay 1e-2 \
--fp16 \
--local_rank=0 \
--sparse-mode alternating \
--checkpoint-activations \
--deepspeed-activation-checkpointing \
--deepspeed \
--deepspeed_config ../src/deepspeed_config/gpt3_xl_sparse_2048.json \
"
run_cmd="USE_DEEPSPEED=1 python3.7 ../pretrain_gpt3.py $@ ${gpt_options}"
echo ${run_cmd}
eval ${run_cmd}
set +x```
but I have error:
```$ ./deepspeed_xl_runormas_v14_130k_finetune.sh
USE_DEEPSPEED=1 python3.7 ../pretrain_gpt3.py --train-data-path /home/mdomrachev/aspect-extractor/entities_normalizer/RuNormAS-solution/data/public_test_v13/files.list --max-files-per-process 10 --logging-dir=/mnt/de420497-d399-4936-af66-6bddbd9bfd33/Data/aspect_miner/demo/ --load /mnt/de420497-d399-4936-af66-6bddbd9bfd33/Data/aspect_miner/mp_rank_00_model_states.pt --save /mnt/de420497-d399-4936-af66-6bddbd9bfd33/Data/aspect_miner/demo/ --tokenizer-path sberbank-ai/rugpt3xl --no-load-optim --finetune --cache-prefix p16 --save-interval 5000 --log-interval 100 --model-parallel-size 1 --num-layers 24 --hidden-size 2048 --num-attention-heads 16 --batch-size 1 --seq-length 2048 --max-position-embeddings 2048 --train-iters 20000 --distributed-backend nccl --lr 0.000015 --warmup 0.0 --lr-decay-style constant --weight-decay 1e-2 --fp16 --local_rank=0 --sparse-mode alternating --checkpoint-activations --deepspeed-activation-checkpointing --deepspeed --deepspeed_config ../src/deepspeed_config/gpt3_xl_sparse_2048.json
using world size: 1 and model-parallel size: 1
> using dynamic loss scaling
> initializing model parallel with size 1
[2021-05-29 00:53:48,161] [INFO] [checkpointing.py:629:_configure_using_config_file] {'partition_activations': False, 'contiguous_memory_optimization': False, 'cpu_checkpointing': False, 'number_checkpoints': None, 'synchronize_checkpoint_boundary': False, 'profile': False}
Pretrain GPT3 model
arguments:
attention_dropout ............ 0.1
num_attention_heads .......... 16
hidden_size .................. 2048
intermediate_size ............ None
num_layers ................... 24
layernorm_epsilon ............ 1e-05
hidden_dropout ............... 0.1
max_position_embeddings ...... 2048
vocab_size ................... 30522
deep_init .................... False
make_vocab_size_divisible_by . 8
cpu_optimizer ................ False
cpu_torch_adam ............... False
sparse_mode .................. alternating
fp16 ......................... True
fp32_embedding ............... False
fp32_layernorm ............... False
fp32_tokentypes .............. False
fp32_allreduce ............... False
hysteresis ................... 2
loss_scale ................... None
loss_scale_window ............ 1000
min_scale .................... 1
batch_size ................... 1
weight_decay ................. 0.01
checkpoint_activations ....... True
checkpoint_num_layers ........ 1
deepspeed_activation_checkpointing True
clip_grad .................... 1.0
train_iters .................. 20000
log_interval ................. 100
logging_dir .................. /mnt/de420497-d399-4936-af66-6bddbd9bfd33/Data/aspect_miner/demo/
exit_interval ................ None
seed ......................... 1234
reset_position_ids ........... False
reset_attention_mask ......... False
lr_decay_iters ............... None
lr_decay_style ............... constant
lr ........................... 1.5e-05
min_lr ....................... 1e-06
warmup ....................... 0.0
save ......................... /mnt/de420497-d399-4936-af66-6bddbd9bfd33/Data/aspect_miner/demo/
save_interval ................ 5000
no_save_optim ................ False
no_save_rng .................. False
load ......................... /mnt/de420497-d399-4936-af66-6bddbd9bfd33/Data/aspect_miner/mp_rank_00_model_states.pt
no_load_optim ................ True
log_memory ................... False
no_load_rng .................. False
load_huggingface ............. None
export_huggingface ........... None
huggingface_double_pos_embeddings False
load_tag .....................
cache_prefix ................. p16
finetune ..................... True
resume_dataloader ............ False
distributed_backend .......... nccl
local_rank ................... 0
master_port .................. 6000
eval_batch_size .............. None
eval_iters ................... 100
eval_interval ................ 1000
eval_seq_length .............. None
eval_max_preds_per_seq ....... None
overlapping_eval ............. 32
cloze_eval ................... False
eval_hf ...................... False
load_openai .................. False
temperature .................. 1.0
top_p ........................ 0.0
top_k ........................ 0
out_seq_length ............... 256
tg_token_name ................ token.txt
model_parallel_size .......... 1
shuffle ...................... False
train_data ................... None
use_npy_data_loader .......... False
train_data_path .............. /home/mdomrachev/aspect-extractor/entities_normalizer/RuNormAS-solution/data/public_test_v13/files.list
val_data_path ................
test_data_path ...............
input_data_sizes_file ........ sizes.txt
delim ........................ ,
text_key ..................... sentence
eval_text_key ................ None
valid_data ................... None
split ........................ 1000,1,1
test_data .................... None
overwrite_cache .............. False
lazy_loader .................. False
loose_json ................... False
presplit_sentences ........... False
num_workers .................. 2
tokenizer_path ............... sberbank-ai/rugpt3xl
cache_dir .................... None
use_tfrecords ................ False
seq_length ................... 2048
max_files_per_process ........ 10
max_preds_per_seq ............ None
path ......................... None
tokenizer_name ............... sberbank-ai/rugpt3xl
output_dir ................... None
part ......................... train
train_part_name .............. train
data_parts ................... generic,named
answer_sep ................... <answer>
window_size .................. 0
start_sep .................... <start>
end_sep ...................... <end>
save_preds_path .............. ../test_pred/
do_sample .................... None
num_beams .................... None
loss_only_norm ............... False
line_by_line ................. False
deepspeed .................... True
deepspeed_config ............. ../src/deepspeed_config/gpt3_xl_sparse_2048.json
deepscale .................... False
deepscale_config ............. None
deepspeed_mpi ................ False
cuda ......................... True
rank ......................... 0
world_size ................... 1
dynamic_loss_scale ........... True
[2021-05-29 00:53:48,161] [INFO] [checkpointing.py:256:model_parallel_cuda_manual_seed] > initializing model parallel cuda seeds on global rank 0, model parallel rank 0, and data parallel rank 0 with model parallel seed: 3952 and data parallel seed: 1234
Load tokenizer from sberbank-ai/rugpt3xl
Add answer_sep: <answer>
Add start_sep <start>
Add start_sep <end>
Load RuGPT3 Dataset from /home/mdomrachev/aspect-extractor/entities_normalizer/RuNormAS-solution/data/public_test_v13/files.list, 10 files per process
R0/1: Loading dataset /home/mdomrachev/aspect-extractor/entities_normalizer/RuNormAS-solution/data/public_test_v13/files.list
R0/1: Check filelist /home/mdomrachev/aspect-extractor/entities_normalizer/RuNormAS-solution/data/public_test_v13/files.list with root dir /home/mdomrachev/aspect-extractor/entities_normalizer/RuNormAS-solution/data/public_test_v13
Shard size 545 > max_file_load 10, only first 10 files of dataset would be loaded!
R0/1: Shard [0, 10]
R0/1: Loaded 0/10 files
R0/1: Loaded 78 examples, 159744 tokens
> padded vocab (size: 50260) with 4 dummy tokens (new size: 50264)
> end-of-document token: 0
building GPT3 model ...
Use sparse attention with mode alternating
Use alternating sparse & dense attention layers
> number of parameters on model parallel rank 0: 1315737600
Optimizer = FusedAdam
learning rate decaying constant
DeepSpeed is enabled.
[2021-05-29 00:54:03,370] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed info: version=0.3.7, git-hash=unknown, git-branch=unknown
[2021-05-29 00:54:03,380] [INFO] [engine.py:588:_configure_optimizer] Using client Optimizer as basic optimizer
[2021-05-29 00:54:03,380] [INFO] [engine.py:597:_configure_optimizer] DeepSpeed Basic Optimizer = FusedAdam (
Parameter Group 0
betas: (0.9, 0.999)
bias_correction: True
eps: 1e-08
lr: 1.5e-05
weight_decay: 0.01
Parameter Group 1
betas: (0.9, 0.999)
bias_correction: True
eps: 1e-08
lr: 1.5e-05
weight_decay: 0.0
)
Checking ZeRO support for optimizer=FusedAdam type=<class 'apex.optimizers.fused_adam.FusedAdam'>
[2021-05-29 00:54:03,380] [INFO] [engine.py:715:_configure_zero_optimizer] Creating fp16 ZeRO stage 2 optimizer
Using /home/mdomrachev/.cache/torch_extensions as PyTorch extensions root...
Emitting ninja build file /home/mdomrachev/.cache/torch_extensions/utils/build.ninja...
Building extension module utils...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module utils...
Time to load utils op: 0.23281097412109375 seconds
[2021-05-29 00:54:03,924] [INFO] [stage2.py:130:__init__] Reduce bucket size 50000000
[2021-05-29 00:54:03,924] [INFO] [stage2.py:131:__init__] Allgather bucket size 500000000
[2021-05-29 00:54:03,924] [INFO] [stage2.py:132:__init__] CPU Offload: False
Traceback (most recent call last):
File "../pretrain_gpt3.py", line 834, in <module>
main()
File "../pretrain_gpt3.py", line 790, in main
model, optimizer, lr_scheduler = setup_model_and_optimizer(args)
File "../pretrain_gpt3.py", line 190, in setup_model_and_optimizer
dist_init_required=False
File "/home/mdomrachev/aspect-extractor/entities_normalizer/RuNormAS-solution/gpt/lib/python3.7/site-packages/deepspeed/__init__.py", line 118, in initialize
config_params=config_params)
File "/home/mdomrachev/aspect-extractor/entities_normalizer/RuNormAS-solution/gpt/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 181, in __init__
self._configure_optimizer(optimizer, model_parameters)
File "/home/mdomrachev/aspect-extractor/entities_normalizer/RuNormAS-solution/gpt/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 609, in _configure_optimizer
self.optimizer = self._configure_zero_optimizer(basic_optimizer)
File "/home/mdomrachev/aspect-extractor/entities_normalizer/RuNormAS-solution/gpt/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 749, in _configure_zero_optimizer
gradient_accumulation_steps=self.gradient_accumulation_steps())
File "/home/mdomrachev/aspect-extractor/entities_normalizer/RuNormAS-solution/gpt/lib/python3.7/site-packages/deepspeed/runtime/zero/stage2.py", line 264, in __init__
self.device).clone().float().detach())
RuntimeError: CUDA out of memory. Tried to allocate 4.90 GiB (GPU 0; 10.91 GiB total capacity; 4.90 GiB already allocated; 4.85 GiB free; 4.90 GiB reserved in total by PyTorch)
gpu = 1080Ti
triton==0.2.2
deepspeed==0.3.7
transformers==3.5.0
torch=1.7.1
But, if I run this code, it;s ok:
import os
os.environ["USE_DEEPSPEED"] = "1"
from src.xl_wrapper import RuGPT3XL
model = RuGPT3XL.from_pretrained(
model_name_or_path="sberbank-ai/rugpt3xl",
seq_len=512,
weights_path="/mnt/de420497-d399-4936-af66-6bddbd9bfd33/Data/aspect_miner/mp_rank_00_model_states.pt",
deepspeed_config_path="/home/mdomrachev/aspect-extractor/entities_normalizer/RuNormAS-solution/src/deepspeed_config/gpt3_xl_sparse_2048.json"
)
Any parameters in the script that need to be changed?