Reproducibility of Training Results

[SEU-zxj]

Hi @liyingyanUCAS, thank you for sharing this excellent work!

I’m currently trying to reproduce your results and have tested both:

• Running inference using the provided pretrained weights.
• Training the model from scratch using the provided scripts.

I was able to reproduce the same performance using the pretrained weights—thanks for that!
However, when I trained the model myself (following the instructions in the README), I could not match the reported results, particularly regarding the collision rate. Please see the comparison below:

As you can see, the L2 metric aligns well with your reported results, but the collision rate differs significantly from the baseline.

Here are some details about my setup:

• Environment: set up strictly following the README instructions.
• Training command used: ./tools/nusc_my_train.sh law/default 8.

Could you kindly advise if there might be any steps I’m missing or if any additional settings are needed to reproduce the results accurately?

Thanks in advance for your help!

[SEU-zxj]

Below is the packages I installed:

(law) zhengxj@node045:~$ pip list
Package                  Version      Editable project location
------------------------ ------------ -----------------------------------------
absl-py                  0.15.0
addict                   2.4.0
anyio                    3.6.2
argon2-cffi              21.3.0
argon2-cffi-bindings     21.2.0
arrow                    1.2.3
asttokens                2.2.1
astunparse               1.6.3
attrs                    22.2.0
backcall                 0.2.0
beautifulsoup4           4.11.2
black                    24.8.0
bleach                   6.0.0
blessed                  1.20.0
cachetools               5.3.0
ccimport                 0.4.2
certifi                  2023.7.22
cffi                     1.15.1
charset-normalizer       3.1.0
click                    8.1.3
comm                     0.1.2
contourpy                1.0.7
cumm-cu114               0.4.7
cycler                   0.11.0
debugpy                  1.6.6
decorator                5.1.1
defusedxml               0.7.1
descartes                1.1.0
einops                   0.8.1
exceptiongroup           1.1.1
executing                1.2.0
fastjsonschema           2.16.3
filelock                 3.16.1
fire                     0.5.0
flake8                   6.0.0
flatbuffers              1.12
fonttools                4.39.2
fqdn                     1.5.1
fsspec                   2025.3.0
gast                     0.3.3
google-auth              2.16.2
google-auth-oauthlib     0.4.6
google-pasta             0.2.0
gpustat                  1.1.1
grpcio                   1.73.1
h5py                     2.10.0
hf-xet                   1.1.5
huggingface-hub          0.33.2
idna                     3.4
imageio                  2.26.0
importlib-metadata       6.1.0
importlib-resources      5.12.0
iniconfig                2.0.0
ipdb                     0.13.13
ipykernel                6.21.3
ipython                  8.11.0
ipython-genutils         0.2.0
ipywidgets               8.0.4
isoduration              20.11.0
jedi                     0.18.2
Jinja2                   3.1.2
joblib                   1.2.0
jsonpointer              2.3
jsonschema               4.17.3
jupyter                  1.0.0
jupyter_client           8.0.3
jupyter-console          6.6.3
jupyter_core             5.3.0
jupyter-events           0.6.3
jupyter_server           2.5.0
jupyter_server_terminals 0.4.4
jupyterlab-pygments      0.2.2
jupyterlab-widgets       3.0.5
Keras-Preprocessing      1.1.2
kiwisolver               1.4.4
lark                     1.1.5
llvmlite                 0.31.0
lyft-dataset-sdk         0.0.8
Markdown                 3.4.1
markdown-it-py           3.0.0
MarkupSafe               2.1.2
matplotlib               3.5.2
matplotlib-inline        0.1.6
mccabe                   0.7.0
mdurl                    0.1.2
mistune                  2.0.5
mmcls                    0.25.0
mmcv-full                1.4.0
mmdet                    2.14.0
mmdet3d                  0.17.1       /opt/nvme0/zhengxj/projects/mmdetection3d
mmengine                 0.10.7
mmsegmentation           0.14.1
mypy-extensions          1.0.0
nbclassic                0.5.3
nbclient                 0.7.2
nbconvert                7.2.10
nbformat                 5.7.3
nest-asyncio             1.5.6
networkx                 2.2
ninja                    1.11.1
notebook                 6.5.3
notebook_shim            0.2.2
numba                    0.48.0
numpy                    1.19.5
nuscenes-devkit          1.1.10
nvidia-ml-py             12.570.86
oauthlib                 3.2.2
onnxruntime              1.8.1
opencv-python            4.7.0.72
opt-einsum               3.3.0
packaging                23.0
pandas                   1.4.4
pandocfilters            1.5.0
parso                    0.8.3
pathspec                 0.11.1
pccm                     0.4.6
pexpect                  4.8.0
pickleshare              0.7.5
Pillow                   9.4.0
pip                      24.2
pkgutil_resolve_name     1.3.10
platformdirs             3.1.1
plotly                   5.13.1
pluggy                   1.0.0
plyfile                  0.8.1
portalocker              2.7.0
prettytable              3.6.0
prometheus-client        0.16.0
prompt-toolkit           3.0.38
protobuf                 4.23.0
psutil                   5.9.4
ptyprocess               0.7.0
pure-eval                0.2.2
pyasn1                   0.4.8
pyasn1-modules           0.2.8
pybind11                 2.10.4
pycocotools              2.0.7
pycodestyle              2.10.0
pycparser                2.21
pyflakes                 3.0.1
Pygments                 2.14.0
pyparsing                3.0.9
pyquaternion             0.9.9
pyrsistent               0.19.3
pytest                   7.2.2
python-dateutil          2.8.2
python-json-logger       2.0.7
pytz                     2022.7.1
PyWavelets               1.4.1
PyYAML                   6.0
pyzmq                    25.0.1
qtconsole                5.4.1
QtPy                     2.3.0
requests                 2.28.2
requests-oauthlib        1.3.1
rfc3339-validator        0.1.4
rfc3986-validator        0.1.1
rich                     14.0.0
rsa                      4.9
safetensors              0.5.3
scikit-image             0.19.3
scikit-learn             1.2.2
scipy                    1.10.1
Send2Trash               1.8.0
setuptools               59.5.0
Shapely                  1.8.5
six                      1.15.0
sniffio                  1.3.0
soupsieve                2.4
spconv-cu114             2.3.3
stack-data               0.6.2
tenacity                 8.2.2
tensorboard              2.19.0
tensorboard-data-server  0.7.2
tensorboard-plugin-wit   1.8.1
termcolor                1.1.0
terminado                0.17.1
terminaltables           3.1.10
threadpoolctl            3.1.0
tifffile                 2023.3.15
timm                     1.0.16
tinycss2                 1.2.1
tomli                    2.0.1
torch                    1.9.1+cu111
torchaudio               0.9.1
torchvision              0.10.1+cu111
tornado                  6.2
tqdm                     4.65.0
traitlets                5.9.0
trimesh                  2.35.39
typing_extensions        4.13.2
uri-template             1.2.0
urllib3                  1.26.15
wcwidth                  0.2.6
webcolors                1.12
webencodings             0.5.1
websocket-client         1.5.1
Werkzeug                 2.2.3
wheel                    0.44.0
widgetsnbextension       4.0.5
wrapt                    1.12.1
yapf                     0.32.0
zipp                     3.15.0

[erow]

the same issue

[erow]

The model is not stable. The performance shows large variance. I run three experiments with different seeds:

s0
plan_L2_1s:0.3501101616888298
plan_L2_2s:0.6757505663672839
plan_L2_3s:1.1105548016498177
plan_obj_box_col_1s:0.0025395585075210004
plan_obj_box_col_2s:0.0032232857980074234
plan_obj_box_col_3s:0.007130299049514559

s1

plan_L2_1s:0.32627387798214486
plan_L2_2s:0.6500824473904456
plan_L2_3s:1.0900157361362894
plan_obj_box_col_1s:0.0008790779449111154
plan_obj_box_col_2s:0.0019535065442469234
plan_obj_box_col_3s:0.006837273073699424

s2

plan_L2_1s:0.2910265245885939
plan_L2_2s:0.5851400871900039
plan_L2_3s:0.990185115239686
plan_obj_box_col_1s:0.003613987106856808
plan_obj_box_col_2s:0.003565149443250635
plan_obj_box_col_3s:0.006641922358144748

[SEU-zxj]

@erow Hello, I guess most of the E2EAD model holds the same problem.
I used to train SSR twice with same seed, and also found that the model output different results (especial collision rate). And that why I come here.

I am curious that whether or not LAW is stable when we use the same seed? If it is stable, maybe it is OK to use the same seed with LAW for comparison.

[SEU-zxj]

@erow For the same seed (default seed), the results are the same. I have checked.

-------------- Planning --------------
plan_L2_1s:0.30336750453458755
plan_L2_2s:0.6079531940217582
plan_L2_3s:1.0211323340235916
plan_obj_col_1s:0.0
plan_obj_col_2s:4.883766360617308e-05
plan_obj_col_3s:3.255844337443258e-05
plan_obj_box_col_1s:0.0009767532721234617
plan_obj_box_col_2s:0.0017093182262160579
plan_obj_box_col_3s:0.005502376906991494

[erow]

@SEU-zxj So, what's your seed for your fist experiment? Your latest result is better than mine but still worse than the result paper reported.

[liyingyanUCAS]

Yes, we observed slight variations in results across different seeds and machines. Our experiments were conducted using the default seed provided in the repository and ran on NVIDIA A6000 GPUs.

[SEU-zxj]

@erow I do not set seed, I change two lines of code in the config file law/default.py, below is what I have changed:

• Line 45, total_epochs = 12 change to total_epochs = 24
• Line 208, checkpoint_config = dict(interval=total_epochs, max_keep_ckpts=total_epochs) change to checkpoint_config = dict(interval=1, max_keep_ckpts=total_epochs)

Below is my results on 24 epochs, and ckpt at epoch 12 got the best result. The averaged collision rate is close to the results provided by @liyingyanUCAS in README.

Hope that helps you :)

[JingGuABC]

Hey @liyingyanUCAS, Thanks for sharing this excellent work.
I encountered the same issue mentioned above. I’m using the same GPUs as you reported (NVIDIA A6000), with the default configuration file (default.py).
The only difference is that I’m running on two A6000 GPUs instead of the full setup. You can find my results below.
I really appreciate this work and would greatly value any suggestions on how to achieve similar results to those reported in your paper.

-------------- Planning --------------
plan_L2_1s:0.3271467112317147
plan_L2_2s:0.643752899200437
plan_L2_3s:1.0671312904822217
plan_obj_col_1s:0.0
plan_obj_col_2s:4.883766360617308e-05
plan_obj_col_3s:9.767533012329775e-05
plan_obj_box_col_1s:0.0012697792537605002
plan_obj_box_col_2s:0.002344207853096308
plan_obj_box_col_3s:0.00628377953633407

[JingGuABC]

@erow @SEU-zxj Hello there, I try to reproduce the ablation study of using World Model or without World Model. My results give the conclusion that the World Model does help the performance. Did you have the same observation?

[SEU-zxj]

@JingGuABC I remember that if I set the loss_rec's weight (the weight of loss of world model) to zero, I do got a worse result. (I mean I found the world model is helpful).

LAW/projects/mmdet3d_plugin/LAW/LAW.py

Line 36 in b2f6a78

wm_loss_weight=0.2,

Below is my result:

You can find that, the result on the left (use world model), the best metrics is (L2 avg=0.66, Col% avg=0.22%) and the results on the reight (do not use world model), the best metrics is (L2 avg=69, Col% avg=0.23%), which is worse compared with the use world model version.

[JingGuABC]

@SEU-zxj Thanks for sharing this info. Yes, I did the same experiment at the beginning. I use the default learning rate 5e-5. The world model seems helpful (just slightly better). However, it's not fair; you remove one loss, but still use the same learning rate, which can decrease the update step. So the second experiment, I increased the learning rate to 7e-5, and I got an even better result than the one with the world model.... I didn't set the 7e-5 to the original LAW. But from your result, I would say the improvement is quite small.

[SEU-zxj]

@JingGuABC for the two experiment above, I use lr=5e-5 (I do not modify the config.yml).

And if you worried about the improvements that the world model brings is quite small, emmmm, I think @liyingyanUCAS's newest work DriveVLA-W0 maybe can convince you that the world model is helpful?

By the way, I think now everyone use world model because it is really a research hot point 😂
For a student like me, the only thing I can do is to follow the trend 😂

If you have more questions, I am glad to discuss here~

[JingGuABC]

@SEU-zxj, I would like to be more careful about their reported results.

Compared to the LAW paper ablation study. My result and your result can't align with their report.

Yes, I agree with you, world model is a hot research topic. Hope we're all good luck~