VM creation fails at "Waiting for SSH to become available..." stage

[rsmithcooper]

When filing a bug please include the following headings, if possible.

Overview of the Issue

I am trying to build a base Ubuntu image and it fails using Packer at the Waiting for SSH to become available... stage. Most notably, at this point, pings succeed very intermittently and SSH does not work. However, this is not the case if I deploy machines manually using similar declarations.

I also noticed that if I accessed the machine through vCenter, it was at the OS install "select language" page even though Packer was still stuck on SSH.

This leads me to believe it is in fact an issue with not being able to connect to the device properly

Reproduction Steps

I adapted what is provided in this repo and stripped it down for testing. Below is what I ended up with.

Packer config below:

### LOCALS ###

locals {
  # Secrets
  admin_user           = aws_secretsmanager("ubuntu_user", "username")
  admin_pass           = aws_secretsmanager("ubuntu_user", "password")
  admin_encrypted_pass = aws_secretsmanager("ubuntu_user", "encrypted_password")
  vcenter_user         = aws_secretsmanager("vcenter_credentials", "vsphere_user")
  vcenter_pass         = aws_secretsmanager("vcenter_credentials", "vsphere_password")
  vcenter_server       = aws_secretsmanager("vcenter_credentials", "vsphere_server")

  # General
  buildtime = formatdate("YYYY-MM-DD hh:mm", timestamp())

  # vSphere
  datacenter      = "test"
  cluster         = "test_cluster"
  datastore       = "test_datastore"
  content_library = "test_library"
  iso             = "ubuntu-22.04.2-live-server-amd64"

  # VM Settings
  vm_name              = "test_ubuntu"
  cpu                  = 4
  cores_per_cpu        = 1
  memory               = 8
  disk_size            = 100
  firmware             = "efi-secure"
  cdrom_type           = "sata"
  disk_controller_type = ["pvscsi"]
  network_1            = "Net_1"
  network_2            = "Net_2"
  network_card         = "vmxnet3"
  guest_os_language    = "en_US"
  guest_os_keyboard    = "us"
  guest_os_timezone    = "UTC"
  guest_os_type        = "ubuntu64Guest"

  # Boot Settings

  http_ip          = null
  http_port_min    = 8000
  http_port_max    = 8099
  ip_wait_timeout  = "20m"
  shutdown_timeout = "15m"
  boot_order       = "disk,cdrom"
  boot_wait        = "5s"
  user_data = {
    "/meta-data" = file("./data/meta-data")
    "/user-data" = templatefile("./data/user-data.pkrtpl.hcl", {
      build_username           = local.admin_user
      build_password           = local.admin_pass
      build_password_encrypted = local.admin_pass
      vm_guest_os_language     = local.guest_os_language
      vm_guest_os_keyboard     = local.guest_os_keyboard
      vm_guest_os_timezone     = local.guest_os_timezone
    })
  }
}

### SOURCE ###

source "vsphere-iso" "linux-ubuntu" {

  # vCenter Credentials
  vcenter_server      = local.vcenter_server
  username            = local.vcenter_user
  password            = local.vcenter_pass
  insecure_connection = true

  # vSphere Settings
  datacenter = local.datacenter
  cluster    = local.cluster
  datastore  = local.datastore

  # VM Settings
  vm_name              = local.vm_name
  guest_os_type        = local.guest_os_type
  firmware             = local.firmware
  CPUs                 = local.cpu
  cpu_cores            = local.cores_per_cpu
  RAM                  = (1024 * local.memory)
  cdrom_type           = local.cdrom_type
  disk_controller_type = local.disk_controller_type
  storage {
    disk_size             = (local.disk_size * 100)
    disk_thin_provisioned = true
  }
  network_adapters {
    network      = local.network_1
    network_card = local.network_card
  }
  network_adapters {
    network      = local.network_2
    network_card = local.network_card
  }
  remove_cdrom         = true
  tools_upgrade_policy = true
  notes                = "Built by Packer on ${local.buildtime}"

  # Removable Media Settings
  iso_paths    = ["${local.content_library}/${local.iso}/${local.iso}.iso"]
  http_content = local.user_data

  # Boot and Provisioning Settings
  http_ip       = local.http_ip
  http_port_min = local.http_port_min
  http_port_max = local.http_port_max
  boot_order    = local.boot_order
  boot_wait     = local.boot_wait
  boot_command = [
    "c<wait>",
    "linux /casper/vmlinuz --- autoinstall ds=\"nocloud-net;seedfrom=http://{{.HTTPIP}}:{{.HTTPPort}}/\"",
    "<enter><wait>",
    "initrd /casper/initrd",
    "<enter><wait>",
    "boot",
    "<enter>"
  ]
  ip_wait_timeout  = local.ip_wait_timeout
  shutdown_command = "echo '${local.admin_pass}' | sudo -S -E shutdown -P now"
  shutdown_timeout = local.shutdown_timeout

  // Communicator Settings and Credentials
  communicator = "ssh"
  ssh_username = local.admin_user
  ssh_password = local.admin_pass
  ssh_port     = 22
  ssh_timeout  = "30m"

  # Template and Content Library Settings
  convert_to_template = true

}

### BUILD ###

build {
  sources = ["source.vsphere-iso.linux-ubuntu"]

}

user-data config below:

autoinstall:
  version: 1
  apt:
    geoip: true
    preserve_sources_list: false
    primary:
      - arches: [amd64, i386]
        uri: http://archive.ubuntu.com/ubuntu
      - arches: [default]
        uri: http://ports.ubuntu.com/ubuntu-ports
  early-commands:
    - sudo systemctl stop ssh
  locale: ${vm_guest_os_language}
  keyboard:
    layout: ${vm_guest_os_keyboard}
  identity:
    hostname: ubuntu-server
    username: ${build_username}
    password: ${build_password}
  ssh:
    install-server: true
    allow-pw: true
  packages:
    - openssh-server
    - open-vm-tools
    - cloud-init
  user-data:
    disable_root: false
    timezone: ${vm_guest_os_timezone}
  late-commands:
    - sed -i -e 's/^#\?PasswordAuthentication.*/PasswordAuthentication yes/g' /target/etc/ssh/sshd_config
    - echo '${build_username} ALL=(ALL) NOPASSWD:ALL' > /target/etc/sudoers.d/${build_username}
    - curtin in-target --target=/target -- chmod 440 /etc/sudoers.d/${build_username}

Packer Version

1.8.6

Plugin Version and Builders

HashiCorp Packer Plugin for VMware vSphere

1.1.1

Please select the builder.

• [] vsphere-iso
• [] vsphere-clone

VMware vSphere Version

Please provide the VMware vSphere version.

Guest Operating System

Ubuntu 22.04 LTS

[SpecialK118]

I'm also stuck with this exact same behavior.
Packer version - 1.9.1

[bidetzz]

@lfc550 I'm also having this issue, what did you change to make it work ? thank you !

[SpecialK118]

In my case, changing the network solved it.

[giridshar841]

In my case, changing the network solved it.

Hi I am stuck at the same issue as well, Where did you change the network? We specify that in vsphere.json correct? is there any other place where we need to change this.
Also, when the VM gets spawned on my vsphere. I am able to ssh to it manually. However, builder/builder credentials are not working for me though.