salt-cloud - Only 2 out of many ec2 providers are queried with -Q #55311
Description
Description of Issue
I've just started with salt-cloud. I've created a single file with a number of providers (we use subaccounts, every provider is an account, all the account have a role allowing admin privileges to the instance). salt-cloud --list-providers is correctly reporting all the providers. When I try salt-cloud -Q only instances from 2 providers, apparently picked randomly among providers, are shown. Basically, if each of the 5 providers has 5 instances, query would show 10 instances instead of 25.
I've tried -l debug and only 2 (sometimes even 1) endpoints are shown, as if the query isn't really trying to list the others. This is the end of the debug:
[...]
[DEBUG ] LazyLoaded proxmox.avail_sizes
[DEBUG ] Using AWS endpoint: ec2.eu-west-1.amazonaws.com
[INFO ] Assuming the role: arn:aws:iam::XXXXXXXXXXX:role/OrganizationAccountAccessRole
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: saltsrv.wallawalla.net
[INFO ] Assuming the role: arn:aws:iam::YYYYYYYYYYYY:role/OrganizationAccountAccessRole
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: saltsrv.wallawalla.net
[DEBUG ] AWS Request: https://ec2.eu-west-1.amazonaws.com/?Action=DescribeInstances&Version=2014-10-01
[DEBUG ] AWS Request: https://ec2.us-east-1.amazonaws.com/?Action=DescribeInstances&Version=2014-10-01
[DEBUG ] AWS Response Status Code: 200
[DEBUG ] LazyLoaded cloud.cache_node_list
[DEBUG ] AWS Response Status Code: 200
[DEBUG ] LazyLoaded cloud.cache_node_list
[DEBUG ] LazyLoaded nested.output
[...]
Interestingly, if I remove one of the provider successfully queried from my list in /etc/salt/cloud.providers.d, another provider takes its place and still just 2 (or 1) providers are queried.
I don't know if it's related, but when I try terminating one of the instances from a working provider Salt successfully find the host, but then it's like losing it again:
[DEBUG ] Using AWS endpoint: ec2.eu-west-1.amazonaws.com [1/1892]
[INFO ] Assuming the role: arn:aws:iam::XXXXXXXXXXX:role/OrganizationAccountAccessRole
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: saltsrv.wallawalla.net
[DEBUG ] AWS Request: https://ec2.eu-west-1.amazonaws.com/?Action=DescribeInstances&Version=2014-10-01
[DEBUG ] AWS Response Status Code: 200
The following virtual machines are set to be destroyed:
ds:
ec2:
DS-BOX12
Proceed? [N/y] y
... proceeding
[INFO ] Destroying in non-parallel mode.
[DEBUG ] Using AWS endpoint: ec2.eu-west-1.amazonaws.com
[INFO ] Assuming the role: arn:aws:iam::XXXXXXXXXXX:role/OrganizationAccountAccessRole
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: saltsrv.wallawalla.net
[DEBUG ] AWS Request: https://ec2.eu-west-1.amazonaws.com/?Action=DescribeInstances&Filter.1.Name=tag%3AName&Filter.1.Value.1=DS-BOX12&Version=2014-10-01
[DEBUG ] AWS Response Status Code: 200
[DEBUG ] Using AWS endpoint: ec2.eu-west-1.amazonaws.com
[DEBUG ] AWS Request: https://ec2.eu-west-1.amazonaws.com/?Action=DescribeInstanceAttribute&Attribute=disableApiTermination&InstanceId=i-0e1aa5c99658ae027&Version=2014-10-01
[DEBUG ] AWS Response Status Code: 400
[ERROR ] AWS Response Status Code and Error: [400 400 Client Error: Bad Request] {'Errors': {'Error': {'Message': "The instance ID 'i-0e1aa5c99658ae027' does not exist", 'Code': 'InvalidI
nstanceID.NotFound'}}, 'RequestID': 'd73588d9-d598-4184-ad97-48b3dd013b9a'}
[DEBUG ] Termination Protection is disabled for DS-BOX12
[DEBUG ] LazyLoaded cloud.fire_event
[DEBUG ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG ] Sending event: tag = salt/cloud/DS-BOX12/destroying; data = {u'instance_id': 'i-0e1aa5c99658ae027', u'_stamp': '2019-11-14T22:32:12.766919', u'name': 'DS-BOX12', u'event': u'dest
roying instance'}
[DEBUG ] Closing IPCMessageClient instance
[INFO ] Renaming DS-BOX12 to DS-BOX12-DELd34461b06183420b91a24744125840fe
[DEBUG ] Using AWS endpoint: ec2.eu-west-1.amazonaws.com
[DEBUG ] AWS Request: https://ec2.eu-west-1.amazonaws.com/?Action=DescribeInstances&Filter.1.Name=tag%3AName&Filter.1.Value.1=DS-BOX12&Version=2014-10-01
[DEBUG ] AWS Response Status Code: 200
[ERROR ] There was an error destroying machines:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/salt/cloud/cli.py", line 210, in run
ret = mapper.destroy(names, cached=True)
File "/usr/lib/python2.7/site-packages/salt/cloud/__init__.py", line 1015, in destroy
ret = self.clouds[fun](name)
File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 3368, in destroy
rename(name, kwargs={'newname': newname}, call='action')
File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 3307, in rename
set_tags(name, {'Name': kwargs['newname']}, call='action')
File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 3124, in set_tags
instance_id = _get_node(name=name, instance_id=None, location=location)['instanceId']
File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 3532, in _get_node
return next(iter(instance_info))
I've tried:
- Splitting the single provider's file into many single files,
- Removing one provider at a time to check if one was causing the issue
- Removing all the profiles (2 very basic profiles, no map files yet)
- Using proper credentials instead of 'use-instance-role-credentials'
I've found this out because I was trying to delete an instance created with salt-cloud but salt couldn't find it, and it did when I've left only that provider in /etc/salt/cloud.providers.d/aws.conf .
Setup
OS: CentOS Linux release 7.7.1908 (Core)
salt-cloud version: salt-cloud 2019.2.2 (Fluorine)
Providers in /etc/salt/cloud.providers.d/aws.conf :
pr1:
driver: ec2
id: 'use-instance-role-credentials'
key: 'use-instance-role-credentials'
role_arn: arn:aws:iam::3262362363262:role/OrganizationAccountAccessRole
private_key: /root/.ssh/id_rsa
keyname: administrator
ssh_username: root
location: eu-west-1
minion:
master: saltsrv.wallawalla.net
pr2:
driver: ec2
id: 'use-instance-role-credentials'
key: 'use-instance-role-credentials'
role_arn: arn:aws:iam::552352352353:role/OrganizationAccountAccessRole
private_key: /root/.ssh/id_rsa
keyname: administrator
location: us-east-1
minion:
master: saltsrv.wallawalla.net
pr3:
driver: ec2
id: 'use-instance-role-credentials'
key: 'use-instance-role-credentials'
role_arn: arn:aws:iam::124241241242:role/OrganizationAccountAccessRole
private_key: /root/.ssh/id_rsa
keyname: administrator
ssh_username: root
location: eu-west-1
minion:
master: saltsrv.wallawalla.net
pr4:
driver: ec2
id: 'use-instance-role-credentials'
key: 'use-instance-role-credentials'
role_arn: arn:aws:iam::1241244124212:role/OrganizationAccountAccessRole
private_key: /root/.ssh/id_rsa
keyname: administrator
ssh_username: root
location: eu-west-1
minion:
master: saltsrv.wallawalla.net
pr5:
driver: ec2
id: 'use-instance-role-credentials'
key: 'use-instance-role-credentials'
role_arn: arn:aws:iam::31233123442:role/OrganizationAccountAccessRole
private_key: /root/.ssh/id_rsa
keyname: administrator
ssh_username: root
location: eu-west-1
minion:
master: saltsrv.wallawalla.net
Steps to Reproduce Issue
salt-cloud --list-providers to check the providers and
salt-cloud -Q to query all the instances. Expected behaviour is to see all instances from all providers listed. Instead, I get only instances from 2 providers.
Versions Report
Salt Version:
Salt: 2019.2.2
Dependency Versions:
cffi: 1.6.0
cherrypy: unknown
dateutil: 2.8.0
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: 2.14
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.5 (default, Aug 7 2019, 00:51:29)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.3.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4
System Versions:
dist: centos 7.7.1908 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-862.3.2.el7.x86_64
system: Linux
version: CentOS Linux 7.7.1908 Core