Question in robotwin2_vla_agent.py

[jiaminghu1994]

def get_action(self, instruction: str, rgbs: np.ndarray) -> np.ndarray: 
    encoded_images = [cv2.imencode('.png', rgb)[1].tobytes() for rgb in rgbs]
    ret = requests.post(
        self.server_url,
        data={"text": instruction, "temperature": 1.0},
        files=[("image", _img) for _img in encoded_images],
    )
    raw_action = ret.json().get('response')
    action_chunk = np.array(raw_action)


    if self.action_horizon > 0 and len(action_chunk) > self.action_horizon:
        action_chunk = action_chunk[:self.action_horizon]
    return action_chunk

Why it only passes instruction and image to policy? What about the robot state such as joint positions?

[jiaminghu1994]

I have trained pi0 policy with dexbotics on absolute space, and there it here. I also found this does not support absolute, so i have to update your code to make it work. RGB image to policy is not correct as well. It seems like this repo is not complete yet.