salt-cloud map partial rebuild doesn't distribute minion keys to master

[BrianSidebotham]

Description of Issue/Question

We have a situation where we're using map files and salt-cloud to create environments of several virtual machines in Proxmox. The complete environment build works as expected. The environments include an ESM (Environment Salt Master) and in the initial build this all works fine - salt-cloud generates minion keys for all of the minions and distributes them to the ESM.

However, if we destroy one of the machines in order to rebuild it and use salt-cloud to rebuild we hit a few problems.

1. The hostname for the ESM cannot be figured out with the current code. I can work around that (I don't really understand yet what that part of the code is meant to be doing in order to get the hostname btw)

2. New minion keys are generated for the missing server, but the server build fails because the ESM rejects the minion's key. This is because salt-cloud doesn't

I can't work around 2. - Does anyone else have this problem, or know a way round this issue?

Setup

A reasonably complex map file.

Steps to Reproduce Issue

Use salt-cloud with a valid map that contains an ESM (environment salt master).
Build the environment successfully.
Terminate one of the VMs
Re-build the environment and salt-cloud will fail as described above.

Versions Report

Salt Version:
           Salt: 2018.11.0-6301-g0a5d2de
 
Dependency Versions:
           cffi: 1.11.5
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
       M2Crypto: 0.28.2
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.1
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Oct 30 2018, 23:45:53)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 18.0.0
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.3.1
 
System Versions:
           dist: centos 7.6.1810 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-957.5.1.el7.x86_64
         system: Linux
        version: CentOS Linux 7.6.1810 Core

[Ch3LL]

ping @saltstack/team-cloud any ideas here?

[gtmanfred]

did you use salt-cloud to delete the previous minions? or did you delete them some other way, and not delete the minions key from salt-key?

[gtmanfred]

Can you provide some steps to actual replicate this?

Thanks.

[BrianSidebotham]

@gtmanfred - I did not use salt-cloud to delete the VM running as a minion for the Environment Salt Master. Instead I simply terminated it at the provider (proxmox) level. Then re-ran the same salt-cloud map.

This results in salt-cloud generating minion keys for the missing VM, but failing to distribute that key to the Environment Salt Master.

Using the below map file snippet as an example:

kvm-centos7-svc:
- salt01-svc.int.wsl:
    clone_name: salt01-svc.int.wsl
    grains:
      wpc_ESM: true
      wpc_env_name: common_services
      wpc_env_proxy: 172.17.240.200:8080
    host: pve110-pplm-mao
    ide2: cloudinit_iso:iso/salt01-svc.int.wsl.iso,media=cdrom
    ip_address: 172.17.0.171
    make_master: true
    minion:
      local_master: true
      master: salt-svc-mao.int.wsl
    name: salt01-svc.int.wsl
    net0: bridge=vmbr2,tag=181
- repo01.int.wsl:
    clone_name: repo01.int.wsl
    grains:
      wpc_env_name: common_services
      wpc_env_proxy: 172.17.240.200:8080
    host: pve110-pplm-mao
    ide2: cloudinit_iso:iso/repo01.int.wsl.iso,media=cdrom
    ip_address: 172.17.0.211
    minion:
      master: 172.17.0.171
    name: repo01.int.wsl
    net0: bridge=vmbr2,tag=181
- proxy01.int.wsl:
    clone_name: proxy01.int.wsl
    grains:
      wpc_env_name: common_services
      wpc_env_proxy: 172.17.240.200:8080
    host: pve110-pplm-mao
    ide2: cloudinit_iso:iso/proxy01.int.wsl.iso,media=cdrom
    ip_address: 172.17.0.212
    minion:
      master: 172.17.0.171
    name: proxy01.int.wsl
    net0: bridge=vmbr2,tag=181

When one of repo01.int.wsl or proxy01.int.wsl is destroyed after initial creation with the salt-cloud map, re-running salt-cloud doesn't successfully build the missing server because the newly-generated minion key for that server is not distributed to the salt master salt01-svc.int.wsl

I could provide some logs if necessary. I expect it's just a current limitation of salt-cloud.

[gtmanfred]

Ahh, i have no idea how the make_master stuff works. maybe @techhat can help.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[waynew]

@Akm0d I know you've been doing some Cloud stuff lately, do you have a handle on this, or should we see if someone in the cloud working group can take a look?

[stale]

Thank you for updating this issue. It is no longer marked as stale.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[sagetherage]

@Akm0d can you please take this to the Cloud Working Group at their next meeting and I will try to get details on the make_master from @saltstack/team-core or perhaps @techhat