Point Cloud: Segmentation, Transforms and Registration

[JemuelStanley47]

1. Point Cloud Segmentation

• Transform RGB and Depth separately to Body Frame
• Mathematically explore if transforming segmentation mask based on RGB would mean the same on Depth frame if on common frame
• If Yes, then do the math to get the segmentation mask at any arbitrary pose in the 3D space based on projection

2. Point Cloud Transforms

• Develop further from Dylan's PointCloud class
• Methods for segmentation, transforms and registration

3. Point Cloud Registration

• Teaser++ for FPFH and local optimization with ICP
• Idea is to check if this can be used for tracking the object in interest

[JemuelStanley47]

Seems like the Depth to RGB frame alignments are performed by image rotations in conq.cameras_utils :

ROTATION_ANGLE = {
    'back_fisheye_image':               0,
    'frontleft_fisheye_image':          -78,
    'frontright_fisheye_image':         -102,
    'frontleft_depth_in_visual_frame':  -78,
    'frontright_depth_in_visual_frame': -102,
    'hand_depth_in_hand_color_frame':   0,
    'hand_depth':                       0,
    'hand_color_image':                 0,
    'left_fisheye_image':               0,
    'right_fisheye_image':              180
}

• Check if this rotation is accurate

Odd that hand_depth and hand_color_image clearly have a rotation difference of +-90 degrees, but this dict says 0 (should be wrong)

• Either change this, and/or report to bosdyn issue
• • Rotation requires the frames to have same shape?. Check if other frames have same shape (DEPTH = RGB) or not
• • Use rotate_image() manually, instead of calling image_to_opencv()

def rotate_image(img, angle):
    img = np.asarray(Image.fromarray(img).rotate(angle, expand=True))
    return img 

• • Perform 3-point calibration to align the depth to camera frame (affine transformation)

[JemuelStanley47]

These are the only available image sources, can't get depth_in_visual_frame directly for hand_frame

RGB_SOURCES = [
    'hand_color_image',
    'back_fisheye_image',
    'frontleft_fisheye_image',
    'frontright_fisheye_image',
    'left_fisheye_image',
    'right_fisheye_image',
]

DEPTH_SOURCES = [
    'hand_depth',
    'frontleft_depth_in_visual_frame',
    'frontright_depth_in_visual_frame',
]

[JemuelStanley47]

Rotated depth image to match RGB orientation using rotate_image

However, affine transformation is required. Need to perform 3-point calibration

Since FOV of Depth << FOV of RGB :

• Crop RGB according to depth frame for segmentation purposes
  OR
• Align RGB to Depth frame (crop out rest) -> Easier to convert to pointcloud
  OR
• Project depth aligned frame with RGB intrinsic?
  OR
• Transform RGB and Depth individually to body frame without alignment and then segment as the combined PC will be in a common frame?

[JemuelStanley47]

Identified hand_depth_in_hand_color_frame image source:

name: "hand_depth_in_hand_color_frame"
cols: 640
rows: 480
pinhole {
  intrinsics {
    focal_length {
      x: 552.02910121610671
      y: 552.02910121610671
    }
    principal_point {
      x: 320
      y: 240
    }
  }
}
image_type: IMAGE_TYPE_DEPTH
pixel_formats: PIXEL_FORMAT_DEPTH_U16
image_formats: FORMAT_RAW

Can also use `hand_color_in_hand_depth_frame"

name: "hand_color_in_hand_depth_frame"
cols: 224
rows: 171
pinhole {
  intrinsics {
    focal_length {
      x: 211.49571228027344
      y: 211.49571228027344
    }
    principal_point {
      x: 110.83219146728516
      y: 87.851737976074219
    }
  }
}
image_type: IMAGE_TYPE_VISUAL
pixel_formats: PIXEL_FORMAT_RGB_U8
image_formats: FORMAT_JPEG
image_formats: FORMAT_RAW

Aligned Depth and RGB Works fine.