From df0821871fe72b3344c6179955f1871a2c0b8bba Mon Sep 17 00:00:00 2001 From: Keita Watanabe Date: Sun, 26 Jan 2025 10:49:32 +0900 Subject: [PATCH] pin datasets library version to avoid dependency conflict Installing llama nxd dependencies with `requirement.txt` on the latest Neuron SDK environment causes following dependency conflict error: ``` ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. neuronx-cc 2.16.372.0+4a9b2326 requires requests<2.32.0, but you have requests 2.32.3 which is incompatible. ``` This `requests` version requirement comes from `datasets` library. ``` pip install requests==2.32.0 Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com Collecting requests==2.32.0 Downloading requests-2.32.0-py3-none-any.whl.metadata (4.6 kB) Requirement already satisfied: charset-normalizer<4,>=2 in /fsx/ubuntu/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from requests==2.32.0) (3.4.1) Requirement already satisfied: idna<4,>=2.5 in /fsx/ubuntu/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from requests==2.32.0) (3.10) Requirement already satisfied: urllib3<3,>=1.21.1 in /fsx/ubuntu/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from requests==2.32.0) (1.26.20) Requirement already satisfied: certifi>=2017.4.17 in /fsx/ubuntu/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from requests==2.32.0) (2024.12.14) WARNING: The candidate selected for download or install is a yanked version: 'requests' candidate (version 2.32.0 at https://files.pythonhosted.org/packages/24/e8/09e8d662a9675a4e4f5dd7a8e6127b463a091d2703ed931a64aa66d00065/requests-2.32.0-py3-none-any.whl (from https://pypi.org/simple/requests/) (requires-python:>=3.8)) Reason for being yanked: Yanked due to conflicts with CVE-2024-35195 mitigation Downloading requests-2.32.0-py3-none-any.whl (63 kB) Installing collected packages: requests Attempting uninstall: requests Found existing installation: requests 2.32.3 Uninstalling requests-2.32.3: Successfully uninstalled requests-2.32.3 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. datasets 3.1.0 requires requests>=2.32.2, but you have requests 2.32.0 which is incompatible. neuronx-cc 2.16.372.0+4a9b2326 requires requests<2.32.0, but you have requests 2.32.0 which is incompatible. ``` Thus we need to install ` datasets==3.0.2` (the latest which satisfies the constraint, released in Oct. 2024) instead of the latest. --- examples/training/llama/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/training/llama/requirements.txt b/examples/training/llama/requirements.txt index d05b079..208ef03 100644 --- a/examples/training/llama/requirements.txt +++ b/examples/training/llama/requirements.txt @@ -1,5 +1,5 @@ transformers==4.31.0 regex tensorboard -datasets +datasets==3.0.2 sentencepiece