git clone private repo breaks during devcontainer build but works afterwards

[faximan]

In my config.json

  "scripts": [
    "if [ ! -d $HOME/.dotfiles ]; then git clone https://github.com/faximan/.dotfiles.git $HOME/.dotfiles; fi",
    "git -C $HOME/.dotfiles pull"
  ]

This is a private GitHub repo.

When I rebuild my devcontainer (which also sets up this feature), this script runs and blocks on Username for github.com.....

However, after I ctrl+c and allow the devcontainer to fully start, and then manually run apply-profile, it works without manual input.

I don't know what is different then? I do have an ssh key registered with GitHub, but I don't think that's used with https? Is it because the GitHub extension has been initialized?