Does this have anything to do with the memory issue? #4
Description
`bash scripts/cli_sat.sh --from_pretrained ./checkpoints/MSAGPT --input-source /home/mca/MP_PSP/myenv/env_name/MSAGPT/INPUT --output-path /home/mca/MP_PSP/myenv/env_name/MSAGPT/output --max-gen-length 64`
> NCCL_DEBUG=VERSION NCCL_IB_DISABLE=0 NCCL_NET_GDR_LEVEL=2 CUDA_LAUNCH_BLOCKING=0 torchrun --nproc_per_node 1 --master_port=19865 /home/mca/MP_PSP/myenv/env_name/MSAGPT/cli_sat.py --bf16 --skip-init --mode finetune --rotary-embedding-2d --seed 12345 --sampling-strategy BaseStrategy --max-gen-length 128 --min-gen-length 0 --num-beams 4 --length-penalty 1.0 --no-repeat-ngram-size 0 --multiline_stream --temperature 0.8 --top_k 0 --top_p 0.9 --from_pretrained ./checkpoints/MSAGPT --input-source /home/mca/MP_PSP/myenv/env_name/MSAGPT/INPUT --output-path /home/mca/MP_PSP/myenv/env_name/MSAGPT/output --max-gen-length 64
> [2025-01-21 11:21:52,777] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
> [2025-01-21 11:21:54,854] [WARNING] No training data specified
> [2025-01-21 11:21:54,855] [WARNING] No train_iters (recommended) or epochs specified, use default 10k iters.
> [2025-01-21 11:21:54,855] [INFO] using world size: 1 and model-parallel size: 1
> [2025-01-21 11:21:54,855] [INFO] > padded vocab (size: 100) with 28 dummy tokens (new size: 128)
> [2025-01-21 11:21:54,855] [INFO] [RANK 0] > initializing model parallel with size 1
> [2025-01-21 11:21:54,856] [INFO] [comm.py:652:init_distributed] cdb=None
> [2025-01-21 11:21:54,856] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 1
> [2025-01-21 11:21:54,857] [INFO] [checkpointing.py:1004:_configure_using_config_file] {'partition_activations': False, 'contiguous_memory_optimization': False, 'cpu_checkpointing': False, 'number_checkpoints': None, 'synchronize_checkpoint_boundary': False, 'profile': False}
> [2025-01-21 11:21:54,857] [INFO] [checkpointing.py:1125:configure] Activation Checkpointing Information
> [2025-01-21 11:21:54,857] [INFO] [checkpointing.py:1126:configure] ----Partition Activations False, CPU CHECKPOINTING False
> [2025-01-21 11:21:54,857] [INFO] [checkpointing.py:1127:configure] ----contiguous Memory Checkpointing False with 6 total layers
> [2025-01-21 11:21:54,857] [INFO] [checkpointing.py:1128:configure] ----Synchronization False
> [2025-01-21 11:21:54,857] [INFO] [checkpointing.py:1129:configure] ----Profiling time in checkpointing False
> [2025-01-21 11:21:54,857] [INFO] [checkpointing.py:229:model_parallel_cuda_manual_seed] > initializing model parallel cuda seeds on global rank 0, model parallel rank 0, and data parallel rank 0 with model parallel seed: 15063 and data parallel seed: 12345
> [2025-01-21 11:21:54,857] [INFO] [RANK 0] building MSAGPT model ...
> [2025-01-21 11:21:54,857] [INFO] [checkpointing.py:229:model_parallel_cuda_manual_seed] > initializing model parallel cuda seeds on global rank 0, model parallel rank 0, and data parallel rank 0 with model parallel seed: 15063 and data parallel seed: 12345
> [2025-01-21 11:21:55,104] [INFO] [RANK 0] > number of parameters on model parallel rank 0: 2860508544
> [2025-01-21 11:21:56,144] [INFO] [RANK 0] CUDA out of memory. Tried to allocate 68.00 MiB (GPU 0; 5.79 GiB total capacity; 5.30 GiB already allocated; 84.75 MiB free; 5.40 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
> [2025-01-21 11:21:56,144] [INFO] [RANK 0] global rank 0 is loading checkpoint ./checkpoints/MSAGPT/1/mp_rank_00_model_states.pt
> [2025-01-21 11:21:58,535] [INFO] [RANK 0] > successfully loaded ./checkpoints/MSAGPT/1/mp_rank_00_model_states.pt
> Traceback (most recent call last):
> File "/home/mca/MP_PSP/myenv/env_name/MSAGPT/cli_sat.py", line 44, in <module>
> model = model.to('cuda')
> ^^^^^^^^^^^^^^^^
> File "/home/mca/anaconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1145, in to
> return self._apply(convert)
> ^^^^^^^^^^^^^^^^^^^^
> File "/home/mca/anaconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 797, in _apply
> module._apply(fn)
> File "/home/mca/anaconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 797, in _apply
> module._apply(fn)
> File "/home/mca/anaconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 797, in _apply
> module._apply(fn)
> [Previous line repeated 2 more times]
> File "/home/mca/anaconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 820, in _apply
> param_applied = fn(param)
> ^^^^^^^^^
> File "/home/mca/anaconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1143, in convert
> return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 68.00 MiB (GPU 0; 5.79 GiB total capacity; 5.30 GiB already allocated; 84.75 MiB free; 5.40 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
> ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 6700) of binary: /home/mca/anaconda3/bin/python
> Traceback (most recent call last):
> File "/home/mca/anaconda3/bin/torchrun", line 8, in <module>
> sys.exit(main())
> ^^^^^^
> File "/home/mca/anaconda3/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
> return f(*args, **kwargs)
> ^^^^^^^^^^^^^^^^^^
> File "/home/mca/anaconda3/lib/python3.11/site-packages/torch/distributed/run.py", line 794, in main
> run(args)
> File "/home/mca/anaconda3/lib/python3.11/site-packages/torch/distributed/run.py", line 785, in run
> elastic_launch(
> File "/home/mca/anaconda3/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
> return launch_agent(self._config, self._entrypoint, list(args))
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/mca/anaconda3/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
> raise ChildFailedError(
> torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
> ============================================================
> /home/mca/MP_PSP/myenv/env_name/MSAGPT/cli_sat.py FAILED
> ------------------------------------------------------------
> Failures:
> <NO_OTHER_FAILURES>
> ------------------------------------------------------------
> Root Cause (first observed failure):
> [0]:
> time : 2025-01-21_11:22:01
> host : mca-lab6
> rank : 0 (local_rank: 0)
> exitcode : 1 (pid: 6700)
> error_file: <N/A>
> traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
> ============================================================
nvidia-smi
> Tue Jan 21 11:24:48 2025
> +---------------------------------------------------------------------------------------+
> | NVIDIA-SMI 535.183.01 Driver Version: 535.183.01 CUDA Version: 12.2 |
> |-----------------------------------------+----------------------+----------------------+
> | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
> | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
> | | | MIG M. |
> |=========================================+======================+======================|
> | 0 NVIDIA GeForce GTX 1660 Ti Off | 00000000:01:00.0 On | N/A |
> | 0% 48C P8 12W / 120W | 219MiB / 6144MiB | 3% Default |
> | | | N/A |
> +-----------------------------------------+----------------------+----------------------+
>
> +---------------------------------------------------------------------------------------+
> | Processes: |
> | GPU GI CI PID Type Process name GPU Memory |
> | ID ID Usage |
> |=======================================================================================|
> | 0 N/A N/A 2307 G /usr/lib/xorg/Xorg 92MiB |
> | 0 N/A N/A 2435 G ...libexec/gnome-remote-desktop-daemon 1MiB |
> | 0 N/A N/A 2473 G /usr/bin/gnome-shell 70MiB |
> | 0 N/A N/A 4776 G ...seed-version=20250119-180455.285000 51MiB |
> +---------------------------------------------------------------------------------------+
This is my INPUT file
7pno_D:GSGSGSGSGTNSLLNLRSRLAAKAAKEAASSNSENLYFQ---SGGTRLTNSLLNLRSRLAAKAAKEAASSNAT------STSGGTRLTNSLLNLRSRLAAKAIKEST----------