Background:
Vscode/devcontainer framework is executing a user mapping, so that the uid of the ebcl user in the container matches the uid on the host. This also updates the permissions of the home folder: https://github.com/devcontainers/cli/blob/7178cebfa88e070439903fd975b7602e7b3f6dc7/scripts/updateUID.Dockerfile#L31
On my machine this takes about 1 minute. I think reason is that it creates an additional yocto layer of about 1.5GB, which is the size of the home folder.
Possible solution: remove the rust toolchain or move the rust toolchain to some other place in the container.