Lanes coordinate systems in nuscenes

[samuruph]

Hello, really impressive work!

I was wondering whether you compute BEV lanes' poins in real world/lidar coordinates for nuscenes.
Because i would integrate your method with some 3d detection models, and I would like to make sure to have exactly the same coordinate systems (or at least to know which transformation I need to get it).

Thank you very much! :)

[cnzzx]

Thanks for your interest. Our experiments were held on 2D BEV coordinates and there might be no height information. If you want to integrate it into your 3d model, changes are required, and the loss should be modified as well (#5).

[samuruph]

Ok hank you very much. But you are using ego or lidar coordinate system? Ego right?

[cnzzx]

Yes, ego. The final prediction is in the ego coordinate system.

[samuruph]

Do you have some functions to project the lanelines onto the multi-view image planes?
I am transforming the lines from ego to lidar (inverse of lidar2ego0 and then project the points onto the image with lidar2img matrix, but apparently i am not getting good results.
This is an example of the resullt, the z coordinate of course is not correct, but even set it to -1.73 the slope is not properly correct, maybe the transformation matrices I am using are not correct.
Thank you!! :)

[cnzzx]

Maybe there is something wrong with your vis. code. This vis. tool from another repo might be helpful (z is default set to 0).

[samuruph]

I tried to use it for front camera only and this is the result i am getting using lidar2img_rt matrix simply.
As you can see it seems something is not correct. I simply used the matrix provided from your dataset geneartion. it seems to be rotated in some way and also the lines seems to be inclined while they shouldn't.
What can I do wrong ?
as you can see the lines have the correct direction, but it seems an offset is introduced. i am simply projecting the predicted lines with lidar2img_rt and subtracting 1.73 from z coordinate

[cnzzx]

Is this on the nuScenes dataset? I only tried their code on Argoverse 2 and it worked well. Maybe there is something different between these two datasets.

[samuruph]

Yes this is Nuscenes dataset.
From the image I sent, you can see that actually the lines have correct direction (parallel to actual lanes in image), but seems to have a wrong origin/fov.
I could also share with you how I am projecting the lines on the image, just to be sure:

for pred_score_3d, pred_bbox_3d, pred_label_3d, pred_pts_3d in zip(scores_3d[keep], boxes_3d[keep],labels_3d[keep], pts_3d[keep]):
            pred_pts_3d = pred_pts_3d.numpy()
            pts_x = pred_pts_3d[:,0]
            pts_y = pred_pts_3d[:,1]
            plt.plot(pts_x, pts_y, color=colors_plt[pred_label_3d],linewidth=1,alpha=0.8,zorder=-1)
            plt.scatter(pts_x, pts_y, color=colors_plt[pred_label_3d],s=1,alpha=0.8,zorder=-1)
            # project pts_3d on img front view
            lidar2img_rt = img_metas[0]['lidar2img'][0]  # CAM_FRONT
            draw_polyline_ego_on_img(pred_pts_3d, img_front, lidar2img_rt, pred_label_3d, thickness=2)

And here the function taken from the repo you shared me:

def draw_polyline_ego_on_img(polyline_ego, img_bgr, lidar2img, map_class, thickness):
    # if 2-dimension, assume z=z_ref
    if polyline_ego.shape[1] == 2:
        z_ref = -1.73
        z_ref_vec = np.ones((polyline_ego.shape[0], 1)) * z_ref 
        polyline_ego = np.concatenate([polyline_ego, z_ref_vec], axis=1)

    polyline_ego = interp_fixed_dist(line=LineString(polyline_ego), sample_dist=0.2)
    
    uv, depth = points_ego2img(polyline_ego, lidar2img)

    h, w, c = img_bgr.shape

    is_valid_x = np.logical_and(0 <= uv[:, 0], uv[:, 0] < w - 1)
    is_valid_y = np.logical_and(0 <= uv[:, 1], uv[:, 1] < h - 1)
    is_valid_z = depth > 0
    is_valid_points = np.logical_and.reduce([is_valid_x, is_valid_y, is_valid_z])

    if is_valid_points.sum() == 0:
        return
    
    tmp_list = []
    for i, valid in enumerate(is_valid_points):
        
        if valid:
            tmp_list.append(uv[i])
        else:
            if len(tmp_list) >= 2:
                tmp_vector = np.stack(tmp_list)
                tmp_vector = np.round(tmp_vector).astype(np.int32)
                draw_visible_polyline_cv2(
                    copy.deepcopy(tmp_vector),
                    valid_pts_bool=np.ones((len(uv), 1), dtype=bool),
                    image=img_bgr,
                    color=colors_cv2[map_class],
                    thickness_px=thickness,
                    map_class=map_class
                )
            tmp_list = []
    if len(tmp_list) >= 2:
        tmp_vector = np.stack(tmp_list)
        tmp_vector = np.round(tmp_vector).astype(np.int32)
        draw_visible_polyline_cv2(
            copy.deepcopy(tmp_vector),
            valid_pts_bool=np.ones((len(uv), 1), dtype=bool),
            image=img_bgr,
            color=colors_cv2[map_class],
            thickness_px=thickness,
            map_class=map_class,
        )

[cnzzx]

Seemingly you just use the "lidar2img" transformation. However, there might be some augmentation (scaling, typically) that will introduce additional transformation. This should be processed like this:

for lidar2img, img_aug in zip(img_metas[0]['lidar2img'],img_metas[0]['img_aug_matrix']):
    inv_aug = np.linalg.inv(img_aug)
    new_lidar2img = np.dot(inv_aug, lidar2img)

In your sample image, for any point on the prediction, you can find that if you link the point to the origin at the upper left corner, after doubling the length (scaling), it will fit well on the real lane.

[samuruph]

Ok, using the inverse augmentation works better but still I am getting some fov issue or similar. Here you can see the precition (left) and gt (right). The line in front of the ego vehicle is correct, but the other lines have still some issue.
I have set the z coordinates of lines to around -1.50 (the camera heihgt should be around 1.50 m). Maybe it could just be the fact that in reality they do not have the same constant height?
Maybe i have to convert the lines in lidar coordinate systems (ego-->lidar) first? Or they are already in lidar coord. system.
Thank you very much for your help!