Get2DBBox and Get3DBBox unexpected behavior

[tomsilver]

The documentation indicates that Get2DBBox and Get3DBBox should return bounding boxes for objects in view, but they don't seem to be working in the code snippet below. The image rgb_hand.png is:

Code:

from pyrcareworld.envs.bathing_env import BathingEnv

import pytest
import cv2
import numpy as np

env = BathingEnv()
img_size = 512
camera_hand.GetRGB(img_size, img_size)
env.step()
rgb = np.frombuffer(camera_hand.data["rgb"], dtype=np.uint8)
rgb = cv2.imdecode(rgb, cv2.IMREAD_COLOR)
assert rgb.shape == (img_size, img_size, 3)
cv2.imwrite("rgb_hand.png", rgb)

# The sponge should be in view.
bboxes2d = camera_hand.Get2DBBox()
assert bboxes2d is not None and len(bboxes2d) > 0  # FAILS because bboxes2d is None

bboxes3d = camera_hand.Get3DBBox()
assert bboxes3d is not None and len(bboxes3d) > 0  # FAILS because bboxes3d is None

[YoruCathy]

Hi @tomsilver the Get2DBBox does not return anything. The BBOX data will be stored in camera.data["2d_bounding_box"] after stepping. Check this example

RCareWorld/pyrcareworld/pyrcareworld/demo/examples/example_camera.py

Line 53 in 76f0bb9

camera.Get2DBBox(512, 512)

[tomsilver]

Ah my fault, thanks for the quick reply!

[tomsilver]

@YoruCathy I'm now getting BBOX data, but it looks like the sponge isn't getting found. Here's an updated snippet:

    from pyrcareworld.envs.bathing_env import BathingEnv

    import pytest
    import cv2
    import numpy as np

    env = BathingEnv(graphics=True)

    robot = env.GetAttr(221582)
    gripper = env.GetAttr(2215820)
    sponge = env.GetAttr(91846)
    camera_hand = env.GetAttr(654321)
    camera_hand.SetTransform(position=gripper.data['position'], rotation=[0, 0, 0])
    camera_hand.SetParent(2215820)
    env.step()

    # Move the gripper up so that the sponge can be seen.
    robot.IKTargetDoMove(
        position=[-0.6, 0.75, 1.25],
        duration=2,
        speed_based=False,
    )
    robot.WaitDo()
    env.step()

    camera_hand.GetRGB(512, 512)
    env.step()
    rgb = np.frombuffer(camera_hand.data["rgb"], dtype=np.uint8)
    rgb = cv2.imdecode(rgb, cv2.IMREAD_COLOR)
    cv2.imwrite("rgb_hand.png", rgb)

    # The sponge should be in view.
    env.step()
    camera_hand.Get2DBBox()
    env.step()
    bboxes2d = camera_hand.data["2d_bounding_box"]
    assert sponge.id in bboxes2d

The result is

>       assert sponge.id in bboxes2d
E       assert 91846 in {221582: [-790428672.0, -9149440.0, 1619251456.0, 45377528.0], 2215820: [21840744.0, 13951530.0, 60043600.0, 31812738.0]}
E        +  where 91846 = <pyrcareworld.attributes.sponge_attr.SpongeAttr object at 0x7f835cfea3e0>.id

[tomsilver]

Probably related, I'm seeing that GetAmodalMask doesn't recognize the sponge either (camera.data["amodal_mask"] is an array full of 205 values after calling GetAmodalMask with the sponge ID and stepping the environment)

[hyhalexis]

Hi Tom, just wondering if you have found out what the problem is? I am having a similar issue with the cloth in the dressing environment not being recognized by GetAmodalMask.

[YoruCathy]

#111 check this post

[shxingch]

Hi @tomsilver @hyhalexis This issue has been fixed and the bathing scene has been updated!