how to add camera intrinsic parameters to model

[qiangqQIANGQ]

Hello ,

Thank you for releasing DetAny3D — I have successfully deployed deploy.py and am able to perform prompt-based 3D detection on monocular images.

My question is : if I already know the camera intrinsic parameters (K) for my image, how can I supply them as an optional input to the model? Specifically:
• Should I encode them in the prompt (as a natural language string) or modify the source code?
• If I need to modify the source code, could you give a short suggest ?

Thanks very much for your time and help!

[kprokofi]

Hi, maybe I can help you.
The model was trained to predict angles relative to its own predicted intrinsics K_pred. If you use K_gt for angle computation you're changing the reference frame the model wasn't trained for. So, it is a mismatch between the virtual camera space the model was trained on and the real camera space of the GT K matrix. That's why the authors provided "decode_bboxes_virtual_to_real" function from train_utils. However, it is not enough to just use this function instead of decode_bbox in the deploy.py code for visualization.

You need to adapt K matrix for the input image (to SAM) (Assume your K matrix is Numpy Array):

def transform_K(K, sx=1.0, sy=1.0, tx=0.0, ty=0.0):
    """
    Apply pixel transform: (u', v') = (sx*u + tx, sy*v + ty).
    K: 3x3 intrinsics
    Returns new 3x3 intrinsics.
    """
    T = np.array([[sx, 0, tx],
                  [0, sy, ty],
                  [0,  0,  1]])
    return T @ K

        # Apply SAM's resize transform
        # sam_trans.apply_image_torch already resized the image, get the scale factor
        sam_scale = sam_trans.target_length / max(original_size)
        K_after_sam_resize = transform_K(K_original, sx=sam_scale, sy=sam_scale)
        
        # Apply center crop transform
        # crop_hw crops to make dimensions divisible by 14 and centers the crop
        original_after_sam = (int(original_size[0] * sam_scale), int(original_size[1] * sam_scale))
        crop_h, crop_w = before_pad_size
        
        # Calculate crop offsets (center crop)
        crop_offset_h = (original_after_sam[0] - crop_h) // 2
        crop_offset_w = (original_after_sam[1] - crop_w) // 2
        
        K_after_crop = transform_K(K_after_sam_resize, tx=-crop_offset_w, ty=-crop_offset_h)
        
        # Apply padding transform (padding doesn't change K, just shifts context)
        # The model expects K relative to the padded image, but since we pad with zeros,
        # the principal point doesn't need adjustment for the valid image region
        
        K_gt = torch.tensor(K_after_crop, dtype=torch.float32).unsqueeze(0).to('cuda:0')

Next you use:

decoded_bboxes_pred_2d, decoded_bboxes_pred_3d = decode_bboxes_virtual_to_real(ret_dict, cfg, K_gt, ret_dict['pred_K'])

And after that you should use K_gt in other functions:

vertices_2d = project_to_image(vertices_3d, K_gt.squeeze(0)) ....

@jjxjiaxue please, correct me, if I am wrong. Thank you for your work.

[kprokofi]

Also, before model inference (For UniDepth model):

input_dict['gt_intrinsic'] = K_gt

[jjxjiaxue]

Sorry for my late reply.
You can try the steps described here:
#10 (comment)

After that, use this function to decode the predicted bbox:
https://github.com/OpenDriveLab/DetAny3D/blob/main/train_utils.py#L368