Skip to content

Problem about evalutations #54

@liangzzs

Description

@liangzzs

When I ran the following command:bash scripts/eval/r2r.sh a8cheng/navila-llama3-8b-8f 1 0 "0"

I encountered the issue shown below. I asked GPT, and it told me that the model was generating a descriptive text completely unrelated to navigation, instead of a prompt for the next action.

Renderer: NVIDIA GeForce RTX 4090/PCIe/SSE2 by NVIDIA Corporation
OpenGL version: 4.6.0 NVIDIA 535.230.02
Using optional features:
    GL_ARB_ES2_compatibility
    GL_ARB_direct_state_access
    GL_ARB_get_texture_sub_image
    GL_ARB_invalidate_subdata
    GL_ARB_multi_bind
    GL_ARB_robustness
    GL_ARB_separate_shader_objects
    GL_ARB_texture_filter_anisotropic
    GL_ARB_texture_storage
    GL_ARB_texture_storage_multisample
    GL_ARB_vertex_array_object
    GL_KHR_debug
Using driver workarounds:
    no-forward-compatible-core-context
    nv-egl-incorrect-gl11-function-pointers
    no-layout-qualifiers-on-old-glsl
    nv-zero-context-profile-mask
    nv-implementation-color-read-format-dsa-broken
    nv-cubemap-inconsistent-compressed-image-size
    nv-cubemap-broken-full-compressed-image-query
    nv-compressed-block-size-in-bits
I1209 07:27:50.384277 652022 ResourceManager.cpp:234] ResourceManager::loadStage : Not loading semantic mesh
I1209 07:27:50.384284 652022 ResourceManager.cpp:262] ResourceManager::loadStage : start load render asset data/scene_datasets/mp3d/zsNo4HB9uLZ/zsNo4HB9uLZ.glb.
I1209 07:27:50.384285 652022 ResourceManager.cpp:569] ResourceManager::loadStageInternal : Attempting to load stage data/scene_datasets/mp3d/zsNo4HB9uLZ/zsNo4HB9uLZ.glb 
I1209 07:27:50.384295 652022 ResourceManager.cpp:1119] Importing Basis files as BC7 for zsNo4HB9uLZ.glb
W1209 07:27:51.227377 652022 Simulator.cpp:248] :
---
 The active scene does not contain semantic annotations. 
---
I1209 07:27:51.250045 651776 simulator.py:213] Loaded navmesh data/scene_datasets/mp3d/zsNo4HB9uLZ/zsNo4HB9uLZ.navmesh
2025-12-09 07:27:51,250 Initializing task VLN-v0
I1209 07:27:51.582638 652022 SceneGraph.h:93] Created DrawableGroup: 
  0%|                                                                                           | 0/1839 [00:00<?, ?it/s]input frame length 8
/home/lhw/miniconda3/envs/navila/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:392: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.0` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
  warnings.warn(
/home/lhw/miniconda3/envs/navila/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:397: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.9` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.
  warnings.warn(
personality and the environment.
[None]
Traceback (most recent call last):
  File "/home/lhw/NaVILA/evaluation/run.py", line 89, in <module>
    main()
  File "/home/lhw/NaVILA/evaluation/run.py", line 39, in main
    run_exp(**vars(args))
  File "/home/lhw/NaVILA/evaluation/run.py", line 83, in run_exp
    trainer.eval()
  File "/home/lhw/NaVILA/evaluation/vlnce_baselines/navila_trainer.py", line 396, in eval
    self._eval_checkpoint(
  File "/home/lhw/NaVILA/evaluation/vlnce_baselines/navila_trainer.py", line 283, in _eval_checkpoint
    outputs = envs.step(actions)
  File "/home/lhw/NaVILA/habitat-lab/habitat/core/vector_env.py", line 447, in step
    self.async_step(data)
  File "/home/lhw/NaVILA/habitat-lab/habitat/core/vector_env.py", line 430, in async_step
    self.async_step_at(index_env, act)
  File "/home/lhw/NaVILA/habitat-lab/habitat/core/vector_env.py", line 404, in async_step_at
    self._warn_cuda_tensors(action)
  File "/home/lhw/NaVILA/habitat-lab/habitat/core/vector_env.py", line 575, in _warn_cuda_tensors
    for k, v in action.items():
AttributeError: 'NoneType' object has no attribute 'items'
  0%|                                                                                           | 0/1839 [00:00<?, ?it/s]

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions