深度估计时，4090显卡爆显存

[Andyxhw]

~/xhw/StereoCrafter$ python depth_splatting_inference.py --pre_trained_path ./weights/stable-video-diffusion-img2vid-xt-1-1 --unet_path ./weights/DepthCrafter --input_video_path ./source_video/zuqiu.mp4 --output_video_path ./outputs/zuqiu_splatting_results.mp4
/home/migu/anaconda3/envs/2to3/lib/python3.8/site-packages/Forward_Warp-0.0.1-py3.8.egg/Forward_Warp/forward_warp.py:19: SyntaxWarning: "is" with a literal. Did you mean "=="?
/home/migu/anaconda3/envs/2to3/lib/python3.8/site-packages/Forward_Warp-0.0.1-py3.8.egg/Forward_Warp/forward_warp.py:57: SyntaxWarning: "is" with a literal. Did you mean "=="?
/home/migu/anaconda3/envs/2to3/lib/python3.8/site-packages/Forward_Warp-0.0.1-py3.8.egg/Forward_Warp/forward_warp.py:58: SyntaxWarning: "is" with a literal. Did you mean "=="?
/home/migu/anaconda3/envs/2to3/lib/python3.8/site-packages/Forward_Warp-0.0.1-py3.8.egg/Forward_Warp/forward_warp.py:19: SyntaxWarning: "is" with a literal. Did you mean "=="?
/home/migu/anaconda3/envs/2to3/lib/python3.8/site-packages/Forward_Warp-0.0.1-py3.8.egg/Forward_Warp/forward_warp.py:57: SyntaxWarning: "is" with a literal. Did you mean "=="?
/home/migu/anaconda3/envs/2to3/lib/python3.8/site-packages/Forward_Warp-0.0.1-py3.8.egg/Forward_Warp/forward_warp.py:58: SyntaxWarning: "is" with a literal. Did you mean "=="?
Loading pipeline components...: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 28.03it/s]
==> processing video: ./source_video/zuqiu.mp4
==> original video shape: (9577, 1080, 1920, 3)
==> downsampled shape: (9577, 576, 1024, 3), with stride: 1
==> final processing shape: (9577, 576, 1024, 3)
Traceback (most recent call last):
File "depth_splatting_inference.py", line 305, in
Fire(main)
File "/home/migu/anaconda3/envs/2to3/lib/python3.8/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/migu/anaconda3/envs/2to3/lib/python3.8/site-packages/fire/core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/home/migu/anaconda3/envs/2to3/lib/python3.8/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "depth_splatting_inference.py", line 287, in main
video_depth, depth_vis = depthcrafter_demo.infer(
File "depth_splatting_inference.py", line 123, in infer
res = self.pipe(
File "/home/migu/anaconda3/envs/2to3/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/migu/xhw/StereoCrafter/dependency/DepthCrafter/depthcrafter/depth_crafter_ppl.py", line 152, in call
video = video.to(device=device, dtype=self.dtype)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 31.57 GiB (GPU 0; 23.53 GiB total capacity; 0 bytes already allocated; 23.07 GiB free; 0 bytes reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

申请31.57G显存，4090只有24G.请问如何缩小申请的内存？

[ColVerDon]

DepthCrafter will load all frame into GPU Mem when read videos, then send them to VAE video encoder to generate latents.This will costs a lot of GPU Mem if the Video sequence is very long.I read the Video to CPU Mem and send them to Cuda before VAE Encoding ,problem solved!
By the way,I think DepthCrafter memory manage is not that ideal and has much space to improve

[Andyxhw]

DepthCrafter will load all frame into GPU Mem when read videos, then send them to VAE video encoder to generate latents.This will costs a lot of GPU Mem if the Video sequence is very long.I read the Video to CPU Mem and send them to Cuda before VAE Encoding ,problem solved! By the way,I think DepthCrafter memory manage is not that ideal and has much space to improve

I'm not very familiar with Python. Can you tell me the code you modified?or show me your code？

[HahaOptimus2]

def infer_pipe(self, video: str,num_denoising_steps: int,guidance_scale: float,save_folder: str,
window_size: int = 110,process_length: int = 195,overlap: int = 25,max_res: int = 1024,
target_fps: int = 15,seed: int = 42,chunk_size=8,name=None):

    set_seed(seed)
    # pdb.set_trace()
    print("==> processing video: ", video)

    # 创建读取fid
    vid_in = cv2.VideoCapture(video)
    frm_count = int(vid_in.get(cv2.CAP_PROP_FRAME_COUNT))
    original_height = int(vid_in.get(cv2.CAP_PROP_FRAME_HEIGHT))
    original_width = int(vid_in.get(cv2.CAP_PROP_FRAME_WIDTH))
    fps = vid_in.get(cv2.CAP_PROP_FPS) if target_fps == -1 else target_fps

    print("==> original video shape: ", (frm_count, original_width, original_height))
    height = round(original_height / 64) * 64   # 四舍五入
    width = round(original_width / 64) * 64

    if max(height, width) > max_res:
        scale = max_res / max(original_height, original_width)
        height = round(original_height * scale / 64) * 64
        width = round(original_width * scale / 64) * 64
    print("==> processing video shape: ", (frm_count, width, height))

     # 创建写入fid
    save_path = os.path.join(
        save_folder, os.path.splitext(os.path.basename(video))[0])
    os.makedirs(os.path.dirname(save_path), exist_ok=True)
    out = cv2.VideoWriter(save_path + "_depth.mp4", 
        cv2.VideoWriter_fourcc(*"mp4v"),fps, (width, height))

    # 打开pickle，用来临时存储
    pik = open(f'./demo_output/tmp_tensor_{name}.pt','wb+')
    
    # 获取add time IDs
    noise_aug_strength = 0.02
    with torch.inference_mode():
        added_time_ids, timesteps, num_inference, latents_init = self.pipe.DatasPrepare(
            noise_aug_strength=noise_aug_strength,
            dtype=torch.float16,
            batch_size=1,
            num_inference_steps=num_denoising_steps,
            window_size=window_size,
            num_channels_latents=8,
            height=height,
            width=width,
            guidance_scale=guidance_scale
        )
    
    # 创建加权权重
    if overlap > 0:
        weights = torch.linspace(0, 1, overlap, device="cpu")
        weights = weights.view(1, overlap, 1, 1, 1)
    else:
        weights = None

    # 不再读取全部帧，每次读取process_length-overlap / process_length帧视频
    # 准备读取的循环
    # pdb.set_trace()
    # 计算batch数量，向上取整，分配每个batch的idx
    blockN = math.ceil(frm_count / (window_size - overlap))
    read_idx = np.array([int(i * (window_size - overlap)) for i in range(blockN)])
   
    # 处理最后一个batch的数据
    read_idx[1:] = read_idx[1:] + overlap
    if read_idx[-1] >= frm_count:
        read_idx = np.delete(read_idx, -1)
    # 设定各batch的解码长度
    decode_len = np.ones((read_idx.shape[0]),dtype = np.int32)
    decode_len = decode_len * (window_size - overlap)
    decode_len[-1] = frm_count - read_idx[-1] + overlap

    # 预留overlap部分的记录数据
    # pre_frames_list = []    # overlap部分的帧信息
    pre_embeddings = None     # 前batch的embedding部分overlap内容
    pre_video_latents = None  # 前batch的video_latents内容
    pre_latents_noise = None  # 前batch的输出latent内容

    total_time, single_batch_time = 0, 0
    # 逐process len读取视频，送入Pipeline处理
    for iter, idx_start in enumerate(list(read_idx)):
        print(f"-------Processing Batch FrameN {idx_start}/{frm_count}-------")
        count = 0
        start_t = time.time() 
        frames_list = []
        
        read_frmN = window_size if idx_start == 0 else window_size - overlap
        iter_frmN = read_frmN if (frm_count - idx_start) >= read_frmN else frm_count - idx_start

        # if not iter == 0:
        #     frames_list = pre_frames_list[-overlap:]

        while(count < iter_frmN):
            ret, frame = vid_in.read()
            frame = cv2.resize(frame,(width, height),0,0,cv2.INTER_CUBIC)
            frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
            frame = frame.astype(np.float32) / 255.0      # 归一化
            frames_list.append(frame)
            count+=1
        
        #  pdb.set_trace()
        # pre_frames_list = frames_list

        frames = np.array(frames_list)        # 转成Numpy
        # inference the depth map using the DepthCrafter pipeline
        with torch.inference_mode():
            res, pre_embeddings, pre_video_latents = self.pipe.forward_pipe(
                frames,height=height,width=width,decode_chunk_size=chunk_size,
                num_inference_steps=num_denoising_steps,
                window_size=window_size,overlap=overlap,
                added_time_ids=added_time_ids, timesteps=timesteps, 
                latents_init=latents_init,pre_embeddings=pre_embeddings, 
                pre_video_latents=pre_video_latents,pre_latents_noise=pre_latents_noise)
        
        # 后半部分噪声与前半部分交换，用于后续片段的overlap内容使用
        latents_init = torch.cat(           
            [latents_init[:, -overlap:], latents_init[:, :window_size-overlap]], dim=1) 
        # 保留 latents输出，将送回inference
        
        #decode
        self.pipe.vae.to(dtype=torch.float16)
        # 此时的latents_all处在cpu
        # pdb.set_trace()
        latents_de = res[:,:decode_len[iter]].clone()
        if not pre_latents_noise == None:
            latents_de[:, :overlap] = latents_de[:, :overlap] * weights + \
                pre_latents_noise * (1 - weights)
        # 解码
        frames = self.pipe.decode_latents_pipe(latents_de, decode_len[iter], chunk_size)
        torch.cuda.empty_cache()
        # pdb.set_trace()
        # 转换成frame
        framesout = self.pipe.video_processor.postprocess_video(
            video=frames, output_type="np")
        # pdb.set_trace()
        pre_latents_noise = res[:,-overlap:]
        
        # 合并成1通道
        res = framesout[0].sum(-1)/framesout[0].shape[-1]
        pickle.dump(res, pik, protocol=4)

        # # pickle.dump(res, open('./demo_output/tmp_tensor.pt','wb+'), protocol=4)
        
        minV, maxV= 0, 1

        res = (res - minV) / (maxV - minV)

        # pdb.set_trace()
        for i in range(decode_len[iter]):
            frame = (res[i] * 255.0).astype(np.uint8)
            frame = cv2.cvtColor(frame, cv2.COLOR_GRAY2BGR)
            out.write(frame)
        end_t = time.time()
        single_batch_time = end_t - start_t

        print(f"-------Processing Batch FrameN {idx_start} cost time: {single_batch_time} s!-------")
        total_time += single_batch_time

you can replace original "infer" function with this module which inferences only window_size frames and writes all depth datas to pickle file in time.