train_ppo_ray.py: unrecognized arguments

[YuZhiyin]

Hi, thank you for your great work!

I was trying to run the training using the script:

sh examples/scripts/train_reinforce_qwen_ray_riv.sh

However, I encountered the following error:

train_ppo_ray.py: error: unrecognized arguments: --logging_path /CoVo/run_outputs/riv/qwen_riv.jsonl --enable_accuracy_filter --save_output_path /CoVo/run_outputs/riv --intrinsic_reward riv

It seems that the current version of train_ppo_ray.py does not recognize these arguments. Just wondering — could it be that the updated version of train_ppo_ray.py with support for these arguments hasn’t been included in the open-source release yet?

Would you mind checking if that file was accidentally left out?

Thanks again for your amazing work, and looking forward to your reply!

[sastpg]

@YuZhiyin Thank you very much for your feedback and for your interest in our work!

I apologize for the inconvenience caused, and have updated the code now. Please pull the latest changes from the repository, and you can use the script to start training.

The core part of reward computation is in CoVo/covo/openrlhf/trainer/ppo_utils/experience_maker.py -> _get_rewards(self, all_prompts, all_outputs, all_labels) and CoVo/covo/openrlhf/trainer/ppo_utils/score.py.

We also updated the description of parameters logging_path and save_output_path in README. The logging_path is the local directory to save the training logs like rewards, loss, kl divergence, etc. We also suggest to use wandb for logging if your network is OK to access wandb. Please refer to the scripts for details.

If you have any further questions or encounter any other issues, please feel free to ask—I'm always happy to help!

[YuZhiyin]

Hi Kongcheng,

Thank you so much for your detailed and constructive feedback — it has been really helpful!

However, I ran into some errors when trying to start the training script, and it seems to be related to my conda environment. I noticed that the requirements.txt file might be missing some key dependencies (for example, vllm).

Would it be possible for you to share a complete pip list of the environment you're using, including the versions of all relevant packages — especially vllm and flash-attn?

That would be really helpful for me to reproduce your setup. Thanks again for your support!

[sastpg]

@YuZhiyin I'm really sorry for the delayed reply.

The vllm version 0.7.2 or 0.7.3 is OK. The flash-attn==2.7.0.post2.
If you encounter some issues when installing flash-attention due to the network problem, visit the FlashAttention release page and find the correct .whl file that matches your:

Python version (e.g., 3.10)
PyTorch version
CUDA version
Download the package using wget:

wget <URL_TO_WHL>

Install it using pip:

pip install <PACKAGE_NAME>.whl

I provide all the requirements and the version I used for checking:

Package                           Version
--------------------------------- ---------------
absl-py                           2.1.0
accelerate                        1.4.0
aiohttp                           3.9.5
aiohttp-cors                      0.7.0
aiosignal                         1.3.1
airportsdata                      20241001
annotated-types                   0.7.0
antlr4-python3-runtime            4.13.2
anyio                             4.8.0
apex                              0.1
argon2-cffi                       23.1.0
argon2-cffi-bindings              21.2.0
asciitree                         0.3.3
astor                             0.8.1
asttokens                         2.4.1
astunparse                        1.6.3
async-timeout                     4.0.3
attrs                             23.2.0
audioread                         3.0.1
beautifulsoup4                    4.12.3
bitsandbytes                      0.45.2
blake3                            1.0.4
bleach                            6.1.0
blis                              0.7.11
cachetools                        5.3.3
catalogue                         2.0.10
certifi                           2024.7.4
cffi                              1.16.0
charset-normalizer                3.3.2
click                             8.1.7
cloudpathlib                      0.18.1
cloudpickle                       3.0.0
cmake                             3.30.0
colorful                          0.5.6
comm                              0.2.2
compressed-tensors                0.9.1
confection                        0.1.5
contourpy                         1.2.1
cuda-python                       12.5.0
cugraph                           24.4.0
cugraph-dgl                       24.4.0
cugraph-equivariant               24.4.0
cugraph-pyg                       24.4.0
cugraph-service-client            24.4.0
cugraph-service-server            24.4.0
cuml                              24.4.0
cupy-cuda12x                      13.0.0
cycler                            0.12.1
cymem                             2.0.8
Cython                            3.0.10
dask                              2024.1.1
dask-cudf                         24.4.0
dask-expr                         0.4.0
datasets                          3.3.2
debugpy                           1.8.2
decorator                         5.1.1
deepspeed                         0.15.0
defusedxml                        0.7.1
depyf                             0.18.0
dill                              0.3.8
diskcache                         5.6.3
distlib                           0.3.9
distributed                       2024.1.1
distro                            1.9.0
dm-tree                           0.1.8
dnspython                         2.7.0
docker-pycreds                    0.4.0
einops                            0.8.0
email_validator                   2.2.0
entrypoints                       0.4
exceptiongroup                    1.2.1
execnet                           2.1.1
executing                         2.0.1
expecttest                        0.1.3
fastapi                           0.115.8
fastapi-cli                       0.0.7
fasteners                         0.19
fastjsonschema                    2.20.0
fastrlock                         0.8.2
filelock                          3.17.0
flash-attn                        2.7.0.post2
fonttools                         4.53.1
frozenlist                        1.4.1
fsspec                            2024.5.0
gast                              0.6.0
gguf                              0.10.0
gitdb                             4.0.12
GitPython                         3.1.44
google-api-core                   2.24.1
google-auth                       2.32.0
google-auth-oauthlib              0.4.6
googleapis-common-protos          1.68.0
grpcio                            1.62.1
h11                               0.14.0
hf-xet                            1.1.0
hjson                             3.1.0
httpcore                          1.0.7
httptools                         0.6.4
httpx                             0.28.1
huggingface-hub                   0.31.1
hypothesis                        5.35.1
idna                              3.7
igraph                            0.11.6
importlib_metadata                7.1.0
iniconfig                         2.0.0
intel-openmp                      2021.4.0
interegular                       0.3.3
ipykernel                         6.29.5
ipython                           8.21.0
ipython-genutils                  0.2.0
isort                             6.0.0
jedi                              0.19.1
Jinja2                            3.1.6
jiter                             0.8.2
joblib                            1.4.2
json5                             0.9.25
jsonlines                         4.0.0
jsonschema                        4.23.0
jsonschema-specifications         2023.12.1
jupyter_client                    8.6.2
jupyter_core                      5.7.2
jupyter-tensorboard               0.2.0
jupyterlab                        2.3.2
jupyterlab_pygments               0.3.0
jupyterlab-server                 1.2.0
jupytext                          1.16.2
kiwisolver                        1.4.5
kvikio                            24.4.0
langcodes                         3.4.0
language_data                     1.2.0
lark                              1.2.2
latex2sympy2_extended             1.0.9
lazy_loader                       0.4
librosa                           0.10.1
lightning-thunder                 0.2.0.dev0
lightning-utilities               0.11.3.post0
lintrunner                        0.12.5
llguidance                        0.7.19
llvmlite                          0.44.0
lm-format-enforcer                0.10.11
locket                            1.0.0
looseversion                      1.3.0
loralib                           0.1.2
marisa-trie                       1.2.0
Markdown                          3.6
markdown-it-py                    3.0.0
MarkupSafe                        2.1.5
math-verify                       0.6.0
matplotlib                        3.9.1
matplotlib-inline                 0.1.7
mdit-py-plugins                   0.4.1
mdurl                             0.1.2
mistral_common                    1.5.4
mistune                           3.0.2
mkl                               2021.1.1
mkl-devel                         2021.1.1
mkl-include                       2021.1.1
mock                              5.1.0
mpmath                            1.3.0
msgpack                           1.0.8
msgspec                           0.19.0
multidict                         6.0.5
multiprocess                      0.70.16
murmurhash                        1.0.10
nanobind                          2.7.0
nbclient                          0.10.0
nbconvert                         7.16.4
nbformat                          5.10.4
nest-asyncio                      1.6.0
networkx                          3.3
ninja                             1.11.1.1
nltk                              3.9.1
notebook                          6.4.10
numba                             0.61.0
numcodecs                         0.11.0
numpy                             1.26.4
nvfuser                           0.2.6a0+f73ff1b
nvidia-cublas-cu12                12.4.5.8
nvidia-cuda-cupti-cu12            12.4.127
nvidia-cuda-nvrtc-cu12            12.4.127
nvidia-cuda-runtime-cu12          12.4.127
nvidia-cudnn-cu12                 9.1.0.70
nvidia-cudnn-frontend             1.5.1
nvidia-cufft-cu12                 11.2.1.3
nvidia-curand-cu12                10.3.5.147
nvidia-cusolver-cu12              11.6.1.9
nvidia-cusparse-cu12              12.3.1.170
nvidia-cusparselt-cu12            0.6.2
nvidia-dali-cuda120               1.39.0
nvidia-ml-py                      12.570.86
nvidia-modelopt                   0.13.0
nvidia-nccl-cu12                  2.21.5
nvidia-nvimgcodec-cu12            0.2.0.7
nvidia-nvjitlink-cu12             12.4.127
nvidia-nvtx-cu12                  12.4.127
nvidia-pyindex                    1.0.9
nvtx                              0.2.5
nx-cugraph                        24.4.0
oauthlib                          3.2.2
onnx                              1.16.0
openai                            1.63.2
opencensus                        0.11.4
opencensus-context                0.1.3
opencv                            4.7.0
opencv-python-headless            4.11.0.86
opt-einsum                        3.3.0
optimum                           1.24.0
optree                            0.15.0
outlines                          0.1.11
outlines_core                     0.1.26
packaging                         24.0
pandas                            2.2.1
pandocfilters                     1.5.1
parso                             0.8.4
partd                             1.4.2
partial-json-parser               0.2.1.1.post5
peft                              0.14.0
pexpect                           4.9.0
pillow                            10.4.0
pip                               24.1.2
platformdirs                      4.2.2
pluggy                            1.5.0
ply                               3.11
polygraphy                        0.49.12
pooch                             1.8.2
preshed                           3.0.9
prometheus_client                 0.20.0
prometheus-fastapi-instrumentator 7.0.2
prompt_toolkit                    3.0.47
proto-plus                        1.26.0
protobuf                          4.24.4
psutil                            5.9.8
ptyprocess                        0.7.0
pure-eval                         0.2.2
py-cpuinfo                        9.0.0
py-spy                            0.4.0
pyarrow                           19.0.1
pyasn1                            0.6.0
pyasn1_modules                    0.4.0
pybind11                          2.13.1
pybind11_global                   2.13.1
pycocotools                       2.0+nv0.8.0
pycountry                         24.6.1
pycparser                         2.22
pydantic                          2.10.6
pydantic_core                     2.27.2
Pygments                          2.18.0
pylibcugraph                      24.4.0
pylibcugraphops                   24.4.0
pylibraft                         24.4.0
pylibwholegraph                   24.4.0
pynvjitlink                       0.2.3
pynvml                            12.0.0
pyparsing                         3.1.2
pytest                            8.1.1
pytest-flakefinder                1.1.0
pytest-rerunfailures              14.0
pytest-shard                      0.1.2
pytest-xdist                      3.6.1
python-dateutil                   2.9.0.post0
python-dotenv                     1.0.1
python-hostlist                   1.23.0
python-json-logger                3.3.0
python-multipart                  0.0.20
pytorch-triton                    3.0.0+989adb9a2
pytz                              2024.1
PyYAML                            6.0.1
pyzmq                             26.0.3
raft-dask                         24.4.0
rapids-dask-dependency            24.4.0a0
ray                               2.46.0
referencing                       0.35.1
regex                             2024.5.15
requests                          2.32.3
requests-oauthlib                 2.0.0
rich                              13.7.1
rich-toolkit                      0.14.5
rmm                               24.4.0
rouge-score                       0.1.2
rpds-py                           0.19.0
rsa                               4.9
safetensors                       0.5.2
scikit-learn                      1.5.1
scipy                             1.13.1
Send2Trash                        1.8.3
sentencepiece                     0.2.0
sentry-sdk                        2.22.0
setproctitle                      1.3.4
setuptools                        68.2.2
shellingham                       1.5.4
six                               1.16.0
smart-open                        7.0.4
smmap                             5.0.2
sniffio                           1.3.1
some-package                      0.1
sortedcontainers                  2.4.0
soundfile                         0.12.1
soupsieve                         2.5
soxr                              0.3.7
spacy                             3.7.5
spacy-legacy                      3.0.12
spacy-loggers                     1.0.5
srsly                             2.4.8
stack-data                        0.6.3
starlette                         0.45.3
sympy                             1.13.1
tabulate                          0.9.0
tbb                               2021.13.0
tblib                             3.0.0
tensorboard                       2.9.0
tensorboard-data-server           0.6.1
tensorboard-plugin-wit            1.8.1
tensorrt                          10.2.0
terminado                         0.18.1
texttable                         1.7.0
thinc                             8.2.5
threadpoolctl                     3.5.0
thriftpy2                         0.5.0
tiktoken                          0.9.0
tinycss2                          1.3.0
tokenizers                        0.21.0
tomli                             2.0.1
toolz                             0.12.1
torch                             2.5.1
torch-tensorrt                    2.5.0a0
torchaudio                        2.5.1
torchmetrics                      1.6.1
torchvision                       0.20.1
tornado                           6.4
tqdm                              4.66.4
traitlets                         5.9.0
transformers                      4.51.3
transformers-stream-generator     0.0.5
treelite                          4.1.2
triton                            3.1.0
typer                             0.12.3
types-dataclasses                 0.6.6
typing_extensions                 4.12.2
tzdata                            2024.1
ucx-py                            0.37.0
urllib3                           2.0.7
uvicorn                           0.34.0
uvloop                            0.21.0
virtualenv                        20.29.2
vllm                              0.7.2
wandb                             0.19.7
wasabi                            1.1.3
watchfiles                        1.0.4
wcwidth                           0.2.13
weasel                            0.4.1
webencodings                      0.5.1
websockets                        15.0
Werkzeug                          3.0.3
wheel                             0.43.0
wrapt                             1.16.0
xdoctest                          1.0.2
xformers                          0.0.28.post3
xgrammar                          0.1.17
xxhash                            3.5.0
yarl                              1.9.4
zarr                              2.18.2
zict                              3.0.0
zipp                              3.19.0

If the conda environment still has some problems, I recommend using this Docker image to start a container, which includes all the packages needed. Docker image URL: https://city-brain-vendor.oss-cn-hangzhou.aliyuncs.com/yq/math/image/openrlhf%3Av1.1.tar

wget https://city-brain-vendor.oss-cn-hangzhou.aliyuncs.com/yq/math/image/openrlhf%3Av1.1.tar
cd CoVo
docker load -i openrlhf\:v1.1.tar
docker run --name covo --runtime=nvidia --gpus all -it --shm-size="32g" -v /path/to/model:/path/to/model -v $PWD:/workspace openrlhf:v1.1 bash
cd /workspace
sh examples/scripts/xxx.sh

I hope this can solve the problem.

[YuZhiyin]

Thank you so much for the detailed response and for taking the time to share all the package versions and setup instructions! This is extremely helpful 🙏. I’ll give it a try. Thanks again!