export HF_TOKEN for CI

[matthiasdiener]

Description

Fixes CI errors of the type:

+ python test_single_gpu_mnist.py
+ tee -a /workspace/examples.log
Namespace(batch_size=64, test_batch_size=800, epochs=10, lr=0.01, momentum=0.9, dry_run=False, seed=1, use_fp8=False, fp8_recipe='DelayedScaling', use_te=False)
Traceback (most recent call last):
  File "/workspace/examples/jax/mnist/test_single_gpu_mnist.py", line 365, in <module>
    train_and_evaluate(mnist_parser(None))
  File "/workspace/examples/jax/mnist/test_single_gpu_mnist.py", line 182, in train_and_evaluate
    train_ds, test_ds = get_datasets()
                        ^^^^^^^^^^^^^^
  File "/workspace/examples/jax/mnist/test_single_gpu_mnist.py", line 143, in get_datasets
    train_ds = load_dataset("ylecun/mnist", split="train", trust_remote_code=True)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/datasets/load.py", line 2062, in load_dataset
    builder_instance = load_dataset_builder(
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/datasets/load.py", line 1782, in load_dataset_builder
    dataset_module = dataset_module_factory(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/datasets/load.py", line 1664, in dataset_module_factory
    raise e1 from None
  File "/opt/venv/lib/python3.11/site-packages/datasets/load.py", line 1629, in dataset_module_factory
    ).get_module()
      ^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/datasets/load.py", line 978, in get_module
    standalone_yaml_path = cached_path(
                           ^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/datasets/utils/file_utils.py", line 189, in cached_path
    ).resolve_path(url_or_filename)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/hf_file_system.py", line 282, in resolve_path
    repo_and_revision_exist, err = self._repo_and_revision_exist(repo_type, repo_id, revision)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/hf_file_system.py", line 209, in _repo_and_revision_exist
    self._api.repo_info(
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 89, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 2997, in repo_info
    return method(
           ^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 89, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 2861, in dataset_info
    hf_raise_for_status(r)
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_http.py", line 743, in hf_raise_for_status
    raise _format(HfHubHTTPError, message, response) from e
huggingface_hub.errors.HfHubHTTPError: (Request ID: Root=1-699e331f-5cf23c0e11020de41b7da4e5;31f28271-bb56-44bd-a6a5-bb381133f2a3)

429 Too Many Requests: you have reached your 'api' rate limit.
Retry after 85 seconds (0/500 requests remaining in current 300s window).
Url: https://huggingface.co/api/datasets/ylecun/mnist/revision/77f3279092a1c1579b2250db8eafed0ad422088c.
We had to rate limit your IP (216.114.73.26). To continue using our service, create a HF account or login to your existing account, and make sure you pass a HF_TOKEN if you're using the API.
##[error]Process completed with exit code 1.

Note that the HF token provided in the GitHub UI is my personal (readonly) token.

Type of change

• Documentation change (change only to the documentation, either a fix or a new content)
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Infra/Build change
• Code refactoring

Changes

Please list the changes introduced in this PR:

Checklist:

• I have read and followed the contributing guidelines
• The functionality is complete
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[ipanfilo]

Printing secret value is undesired

[matthiasdiener]

It's not printing the value, just true/false.

[ipanfilo]

If the token is not accessible, will it just get empty string or throw an error?

[matthiasdiener]

It will be an empty string: https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/use-secrets#using-secrets-in-a-workflow

If a secret has not been set, the return value of an expression referencing the secret (such as ${{ secrets.SuperSecret }} in the example) will be an empty string.

[ipanfilo]

Is docker command echoed in the log?

[matthiasdiener]

The docker command itself is part of the log, but the token value is not echoed: https://productionresultssa19.blob.core.windows.net/actions-results/c226dddb-5f82-4c0d-b1fc-9d6a13e6dded/workflow-job-run-6aa7f3d9-cf79-5c75-a431-db178093cdd2/logs/job/job-logs.txt?rsct=text%2Fplain&se=2026-02-26T19%3A28%3A52Z&sig=WcVcyZzjaFHyOKzn1edEqhyTyP8uodx7xJChWN2gcWI%3D&ske=2026-02-26T22%3A41%3A33Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2026-02-26T18%3A41%3A33Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2025-11-05&sp=r&spr=https&sr=b&st=2026-02-26T19%3A18%3A47Z&sv=2025-11-05

2026-02-26T16:55:04.8279218Z ##[group]Run docker exec -e HF_TOKEN="$HF_TOKEN" te-runner bash -c "$(cat <<'EOF'
2026-02-26T16:55:04.8279548Z �[36;1mdocker exec -e HF_TOKEN="$HF_TOKEN" te-runner bash -c "$(cat <<'EOF'�[0m
2026-02-26T16:55:04.8279734Z �[36;1m#!/usr/bin/bash�[0m
2026-02-26T16:55:04.8279852Z �[36;1mset -ex -o pipefail�[0m
2026-02-26T16:55:04.8279987Z �[36;1mulimit -c 0 # Disable core dumps�[0m
2026-02-26T16:55:04.8280127Z �[36;1m�[0m
2026-02-26T16:55:04.8280235Z �[36;1m# Check whether the HF_TOKEN is present�[0m
2026-02-26T16:55:04.8280446Z �[36;1mpython -c "import os; print('HF_TOKEN set:', bool(os.environ.get('HF_TOKEN')))"�[0m
2026-02-26T16:55:04.8280639Z �[36;1m�[0m
2026-02-26T16:55:04.8280748Z �[36;1mcd /workspace/examples/pytorch/mnist�[0m
2026-02-26T16:55:04.8281011Z �[36;1mpython main.py 2>&1 | tee /workspace/examples.log�[0m
2026-02-26T16:55:04.8281210Z �[36;1mpython main.py --use-te 2>&1 | tee -a /workspace/examples.log�[0m
2026-02-26T16:55:04.8281444Z �[36;1mpython main.py --use-fp8 2>&1 | tee -a /workspace/examples.log�[0m
2026-02-26T16:55:04.8281608Z �[36;1m�[0m
2026-02-26T16:55:04.8281704Z �[36;1mcd /workspace/examples/jax/mnist�[0m
2026-02-26T16:55:04.8281845Z �[36;1mpip3 install -r requirements.txt�[0m
2026-02-26T16:55:04.8282030Z �[36;1mpython test_single_gpu_mnist.py 2>&1 | tee -a /workspace/examples.log�[0m
2026-02-26T16:55:04.8282281Z �[36;1mpython test_single_gpu_mnist.py --use-te 2>&1 | tee -a /workspace/examples.log�[0m
2026-02-26T16:55:04.8282546Z �[36;1mpython test_single_gpu_mnist.py --use-fp8 2>&1 | tee -a /workspace/examples.log�[0m
2026-02-26T16:55:04.8282739Z �[36;1m�[0m
2026-02-26T16:55:04.8282841Z �[36;1mcd /workspace/examples/jax/encoder�[0m
2026-02-26T16:55:04.8282978Z �[36;1mpip3 install -r requirements.txt�[0m
2026-02-26T16:55:04.8283166Z �[36;1mpython test_single_gpu_encoder.py 2>&1 | tee -a /workspace/examples.log�[0m
2026-02-26T16:55:04.8283425Z �[36;1mpython test_single_gpu_encoder.py --use-fp8 2>&1 | tee -a /workspace/examples.log�[0m
2026-02-26T16:55:04.8283620Z �[36;1mEOF�[0m
2026-02-26T16:55:04.8283709Z �[36;1m)"�[0m
2026-02-26T16:55:04.8289538Z shell: /usr/bin/bash -e {0}
2026-02-26T16:55:04.8289656Z env:
2026-02-26T16:55:04.8289744Z   TEST_LEVEL: 1
2026-02-26T16:55:04.8290157Z   HF_TOKEN: ***
2026-02-26T16:55:04.8290256Z ##[endgroup]
2026-02-26T16:55:04.8837538Z + ulimit -c 0
2026-02-26T16:55:04.8837782Z + python -c 'import os; print('\''HF_TOKEN set:'\'', bool(os.environ.get('\''HF_TOKEN'\'')))'
2026-02-26T16:55:04.8917423Z HF_TOKEN set: True