DM0测试Libero数据集state选择

[shee-gao-2]

  action = DM0._get_response(
          text=task.language,
          images=[
              obs['agentview_image'],           
              obs['robot0_eye_in_hand_image'] 
          ],
          states=obs[''],
          batch_size=1
      )

你们的工作非常棒，但我在用DM0-libero模型测试libero-goal数据集时，不太清楚states该如何输入呢？另外是否有DM0在libero数据集上测试的案例呢。

[zed0630]

@shee-gao-2 感谢你的关注和使用！
实际上，DM0 的推理并不包含与 state 相关的计算逻辑，后者仅仅作为对 batch size 等信息的来源被引入到推理过程中。如果你想知道如何构造包含 state 的请求内容以调用模型推理接口，可以参考 dexbotic-benchmark 中已有的实现：https://github.com/dexmal/dexbotic-benchmark/blob/main/evaluation/evaluator/libero_evaluator.py#L257-L262
关于 DM0 在 libero 数据集上的测试，同样可以参考 dexbotic-benchmark 上的实现：https://github.com/dexmal/dexbotic-benchmark/blob/main/evaluation/configs/libero/example_pi0_libero.yaml

[sheegao]

我自己写了libero测试的程序，调用了DM0推理的接口，但是执行的效果似乎不太正常，能麻烦帮看一下嘛？

# Start evaluation
    total_episodes, total_successes = 0, 0
    with open("/vla/dexbotic/checkpoints/DM0-libero/norm_stats.json", "r") as f:
        norm_state = json.load(f)
    for k, v in norm_state.items():
        if "mean" in v:
            v["mean"] = np.array(v["mean"], dtype=np.float32)
        if "std" in v:
            v["std"] = np.array(v["std"], dtype=np.float32)
    for task_id in tqdm.tqdm(task_indices):
        # Get task
        task = task_suite.get_task(task_id)

        # Get default LIBERO initial states
        initial_states = task_suite.get_task_init_states(task_id)
        
        # Initialize LIBERO environment and task description
        env, task_description = get_libero_env(task, resolution=256)

        DM0 = DM0InferenceConfig(norm_stats=norm_state)
        DM0._load_model()

        # Start episodes
        task_episodes, task_successes = 0, 0
        max_trials = min(cfg.num_trials_per_task, len(initial_states))
        for episode_idx in tqdm.tqdm(range(max_trials)):
            print(f"\nTask: {task_description}")
            log_file.write(f"\nTask: {task_description}\n")

            # Reset environment
            env.reset()

            # Set initial states
            obs = env.set_init_state(initial_states[episode_idx])

            # Setup
            t = 0
            top_view = []
            wrist_view = []
            if cfg.task_suite_name == "libero_spatial":
                max_steps = 220  # longest training demo has 193 steps
            elif cfg.task_suite_name == "libero_object":
                max_steps = 280  # longest training demo has 254 steps
            elif cfg.task_suite_name == "libero_goal":
                max_steps = 300  # longest training demo has 270 steps
            elif cfg.task_suite_name == "libero_10":
                max_steps = 1000  # longest training demo has 505 steps
            elif cfg.task_suite_name == "libero_90":
                max_steps = 400  # longest training demo has 373 steps

            print(f"Starting episode {task_episodes+1}...")
            log_file.write(f"Starting episode {task_episodes+1}...\n")
            while t < max_steps + cfg.num_steps_wait:
                # try:
                # IMPORTANT: Do nothing for the first few timesteps because the simulator drops objects
                # and we need to wait for them to fall
                if t < cfg.num_steps_wait:
                    obs, reward, done, info = env.step(get_libero_dummy_action())
                    t += 1
                    continue

                # # Get preprocessed image
                img, wrist_img = get_libero_image(obs)
    
                # # Save preprocessed image for replay video
                top_view.append(img)
                wrist_view.append(wrist_img)


                def numpy_to_tmp_path(img_np, save_dir="."):
                    os.makedirs(save_dir, exist_ok=True)
                    filename = f"tmp_{uuid.uuid4().hex}.png"
                    path = os.path.join(save_dir, filename)
                    Image.fromarray(img_np).save(path)
                    return path

                img_path = numpy_to_tmp_path(img)
                wrist_path = numpy_to_tmp_path(wrist_img)

                # Query model to get action
                action = DM0._get_response(
                    text=task.language,
                    images=[img_path, wrist_path],
                    batch_size=1
                )

                os.remove(img_path)
                os.remove(wrist_path)     
                # Execute action in environment
                # obs, reward, done, info = env.step(action[0])
                # if done:
                #     task_successes += 1
                #     total_successes += 1
                #     break
                # t += 1
                for step_idx in range(5):  # 执行前 N 步
                    # 获取当前状态
                    delta_action = action[step_idx]
                    obs, reward, done, info = env.step(delta_action)
                    t += 1
                    
                    img, wrist_img = get_libero_image(obs)
                    top_view.append(img)
                    wrist_view.append(wrist_img)

                    if done:
                        task_successes += 1
                        total_successes += 1
                        break
                # except Exception as e:
                #     print(f"Caught exception: {e}")
                #     log_file.write(f"Caught exception: {e}\n")
                #     break

            task_episodes += 1
            total_episodes += 1

            # Save a replay video of the episode
            save_rollout_video(
                top_view,
                wrist_view,
                total_episodes,
                success=done,
                task_description=task_description,
                log_file=log_file,
            )

2026_03_27-10_35_25--episode.1--success.False--task.open_the_middle_drawer_of_the_cabinet.mp4

[zed0630]

@sheegao 基于你提供的代码，我们无法定位到具体的问题。

针对 Libero benchmark，我们也开源了对应的评测工具 Dexbotic-benchmark。你可以在这个工具上测试 DM0 模型，看看结果是否正确。