[Dataset] [ASE] - can't get rgb values of points from semidense_observations.csv.gz

[liuyu19970607]

In our setting ,we want to get the rgb values of the semi dense points. We preprocess the semidense_observations.csv.gz like this, but the rgb value is wrong:

`observe_df = pd.read_csv(observe_file_path, compression='gzip')
point_df = pd.read_csv(point_file_path, compression='gzip')
merged_df = pd.merge(observe_df, point_df, on='uid', how='inner')

points = merged_df[['px_world', 'py_world', 'pz_world']].values
uvs = merged_df[["frame_tracking_timestamp_us", 'u', 'v']].values
uvs[:, 0] = uvs[:, 0] // 1e5
uvs = uvs.astype(int)

rgbs = []
rgb_path_list = os.listdir(rgb_path)
rgb_path_list.sort()
for rgb_name in rgb_path_list:
if not rgb_name.endswith(".jpg"): continue
rgb = cv2.imread(os.path.join(rgb_path, rgb_name))
rgb = cv2.cvtColor(rgb, cv2.COLOR_BGR2RGB)
rgbs.append(rgb)
rgbs = np.stack(rgbs)
colors = rgbs[uvs[:, 0], uvs[:, 1] , uvs[:, 2]]`

looking forward for your reply, thanks

[SeaOtocinclus]

Note that the (u,v) projections saved are for a given camera serial.
https://facebookresearch.github.io/projectaria_tools/docs/data_formats/mps/slam/mps_pointcloud#point-observations
So you have to merge on the camera_serial to match the RGB camera, and if not available get color by projecting the point that is visible in SLAM to the RGB camera.

We have code sample showing how to retrieve the color of the points:

• In C++ - a binary to extract the color of the visible points (https://github.com/facebookresearch/projectaria_tools/tree/main/tools/samples/pointcloud_colorization)
• In Python - a code sample showing you how to use the data structure and know which point is visible in each camera (https://github.com/facebookresearch/projectaria_tools/tree/main/tools/samples/python/mps_semidense_point_visibility)