Visual Servoing: ToDOs

[JemuelStanley47]

• Separate threads for visual pose stream and controller
• Transformation between camera frame and hand frame
• Transformation between hand frame and gravity aligned body frame
• Settle on Object pose convention
• Switch hand pose frame or object pose frame
• Settle on get_object_pose() --> Return (pos,rot) / (pos,quat) / (pos,euler)
• Modify arm move commands' args based on return pose type
• Fix PD controller
• Estimate latency

[JemuelStanley47]

Oversight:

1. Achieves only pose, and not orientation (PBVS accounts for only cartesian pose)

• Align to object/grasp pose (position, orientation)
• Future: Track Delta p_{t}^{t-1} Reference to tracking issue

2. Need to hover with offset along z_axis of grasp/object pose (preferably grasp pose)

• Do the math for hovering over object along z-axis of grasp pose
• Radius of sphere = offset

3. New objective of visual servoing:

• Gripper frame needs to decrease the error between the offset grasp pose on the sphere and current pose
• Might need to use the same controller used to decrease the error between robot current pose and robot measured pose
• When error is within a specific threshold, decrease z linearly to go to true grasp pose

[JemuelStanley47]

TODOs post discussion (11-March-2024):

• Get the centroid of object (roboflowv/ lang-prompted)
• Get waypoints of camera/hand pose based on generated grasp pose (Initial trajectory)
• Best View Point Planning Best View Point Selection #50
• • Objective is to maximize the view of the tool from ray-tracing until a specific threshold(distance between camera and object/grasp pose) is achieved (because further than that)
• • Modify trajectory based on the planner
• Perform normal visual-servoing controller

[JemuelStanley47]

Manipulation check with/without VisualServoing

• Precision and Repeatability check
• • Without visual servoing
• • With Visual servoing (Use AprilTag)
• • Maybe do same experiment with impedance with maximum stiffness?

[JemuelStanley47]

• XMem Object video segmentation
• • https://github.com/hkchengrex/XMem
• • Checkout colab notebook: https://colab.research.google.com/drive/1RXK5QsUo2-CnOiy5AOSjoZggPVHOPh1m?usp=sharing#scrollTo=qYv3kbaQT2w4
• Segment point-cloud based on this segmentation Point Cloud: Segmentation, Transforms and Registration #49