[BUG] Tokenizer download isn't using env_vars in headless installation mode

[keyvaann]

Describe the bug
I'm trying to install the benchmark in headless mode but the tokenizer isn't using the HF_TOKEN set in env_vars and installation fails because of it.

Steps/Code to reproduce bug
Here is my headless play file:

venv_type: uv
install_path: /dgxc-benchmarking/gb300/workloads
slurm_info:
  slurm:
    account: root
    gpu_partition: main
    cpu_partition: main
    gpu_partition_gres: 8
    cpu_partition_gres: null
    node_architecture: aarch64
gpu_type: gb300
node_architecture: aarch64
install_method: local
selected_workloads:
  - pretrain_nemotron4-15b
  - pretrain_nemotron4-340b
  - pretrain_qwen3
  - pretrain_nemotron-h
env_vars:
  HF_TOKEN: hf_..

And I use this command to run it: ./install.sh --play config.yaml -v -d.

I have noticied manually exporting HF_TOKEN to environment variables solves this issue and the script is able to download the tokenizer files.

Expected behavior
The installation will succeed.

Environment details (please complete the following information):

• Environment location: Cloud(Nebius)
• Method of DGXC Benchmarking install: From source with UV
• Run print_env.sh from the project root and paste the results here

Additional context
Here is the output:

Downloading tokenizer files...
(Framework warnings and file progress bars are expected)
2026-01-10 17:00:05,246 - llmb_install.llmb_install.downloads.tokenizer - DEBUG - No HF_TOKEN provided - downloads may be rate limited
2026-01-10 17:00:05,246 - llmb_install.llmb_install.downloads.tokenizer - DEBUG - Processing tokenizer 1/4: Qwen/Qwen3-235B-A22B

[1/4] Qwen/Qwen3-235B-A22B
...
2026-01-10 17:00:17,966 - llmb_install.llmb_install.downloads.tokenizer - DEBUG -   ✓ Offline verification passed
  ✓ Ready for offline use
2026-01-10 17:00:17,966 - llmb_install.llmb_install.downloads.tokenizer - DEBUG - Processing tokenizer 2/4: Qwen/Qwen3-30B-A3B

[2/4] Qwen/Qwen3-30B-A3B
...
2026-01-10 17:00:20,419 - llmb_install.llmb_install.downloads.tokenizer - DEBUG -   ✓ Offline verification passed
  ✓ Ready for offline use
2026-01-10 17:00:20,419 - llmb_install.llmb_install.downloads.tokenizer - DEBUG - Processing tokenizer 3/4: meta-llama/Meta-Llama-3-70B

[3/4] meta-llama/Meta-Llama-3-70B
2026-01-10 17:00:20,419 - llmb_install.llmb_install.downloads.tokenizer - DEBUG -   → Attempting AutoTokenizer.from_pretrained()
2026-01-10 17:00:20,674 - llmb_install.llmb_install.downloads.tokenizer - DEBUG -   ✗ AutoTokenizer.from_pretrained() failed: You are trying to access a gated repo.
Make sure to have access to it at https://huggingface.co/meta-llama/Meta-Llama-3-70B.
401 Client Error. (Request ID: Root=1-696285a4-0e3354fa12398c8b7167231d;6ad9a70f-2869-4631-87a4-b403a299d565)

Cannot access gated repo for url https://huggingface.co/meta-llama/Meta-Llama-3-70B/resolve/main/config.json.
Access to model meta-llama/Meta-Llama-3-70B is restricted. You must have access to it and be authenticated to access it. Please log in.
2026-01-10 17:00:20,674 - llmb_install.llmb_install.downloads.tokenizer - DEBUG -   → Falling back to snapshot_download() with explicit file patterns
README.md: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 36.6k/36.6k [00:00<00:00, 85.8MB/s]
Fetching 9 files:  11%|█████████████████▊                                                                                                                                              | 1/9 [00:00<00:03,  2.23it/s]

Error: 401 Client Error. (Request ID: Root=1-696285a5-34a93ea416cbd52665e28f16;d01506b9-8eec-4f36-a17d-32ba90d00cf0)

Cannot access gated repo for url https://huggingface.co/meta-llama/Meta-Llama-3-70B/resolve/c82494877ce7f6d7d317c56ec081328e382c72fe/config.json.
Access to model meta-llama/Meta-Llama-3-70B is restricted. You must have access to it and be authenticated to access it. Please log in.

---

By submitting this issue, you agree to follow our code of conduct.

[sudostock]

At first glance I think the fix for #26 will fix this as well.