Presence detection for minions behind NAT not working

[brejoc]

Description of Issue/Question

Presence detection for minions behind a NAT is not working. In minions.py we fetch all of the IPv4 addresses of the minions. For minion1 this will be the IP address the master can see. But for minion2 this will be the address the minion gets behind the NAT. Then those addresses are compared to the addresses the master can see from the minions, which will be the address of the NAT for minion2.

I'm not sure how to address this. Any hints welcome!

Setup

minion1 is directly connected to the master. minion2 is behind a NAT.

Steps to Reproduce Issue

Let both minions connect and fire a salt-run manage.alived.

Then you will see this:

$ salt-run manage.alived
- minion1

While this was expected, since both minions are connected:

$ salt-run manage.alived
- minion1
- minion2

Versions Report

Salt Version:
           Salt: 2018.3.0
 
Dependency Versions:
           cffi: 1.5.2
       cherrypy: 3.6.0
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.29.0
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.10
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.13 (default, Jan 11 2017, 10:56:06) [GCC]
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 14.0.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.4
 
System Versions:
           dist: SuSE 12 x86_64
         locale: UTF-8
        machine: x86_64
        release: 4.4.73-5-default
         system: Linux
        version: SUSE Linux Enterprise Server  12 x86_64

But since this was introduced in 2015.8.0, this should be faulty all the way down.

[garethgreenaway]

@saltstack/team-core Any thoughts on this one?

[waynew]

Should manage.alive(d?) not just operate roughly like test.version (or test.ping) where it asks the minion to report back that it's alive?

[brejoc]

@waynew No, alived is different in that regard. A test.ping puts some amount of pressure on the system/network if the installation is really big. So having something that works in a passive way (which alived does) is preferred in this case.

[waynew]

Ah, I see. So it sounds like the problem is really just that the master is missing some bit of information - perhaps there should be another grain set from the minion corresponding to the visible IP from the master?

I'm fairly new to that corner of the world, but given what I just read about alived:

Print a list of all minions that are up according to Salt's presence detection (no commands will be sent to minions)

That, combined with the source that you linked to, leads me to believe that there's a grain that should be there that isn't.

Of course we may not have thought about such a layout when we were building that feature, so we may never have written code to support that. But we should 😀

[DmitryKuzmenko]

@brejoc if you'll append ipv4 grain with the external minion ip address the alived shall work. Is it possible for you to do salt minion2 grains.append ipv4 1.2.3.4 where 1.2.3.4 is the external address of your minion and re-run the manage.alived runner?
I'm not saying it's designed to be used in this way but by the code lookup I bet it will work.

[brejoc]

@DmitryKuzmenko This workaround should indeed work, but might also cause additional problems when relying on that grain for other purposes.

[brejoc]

I just did a test-run with two minions behind the same NAT. Both got the IPv4 address of the NAT as the ipv4 grain. As expected both of them are in the list wen salt-run manage.alived is executed. Even if one of them is turned off. So this is not a viable work-around.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[sagetherage]

@waynew or @DmitryKuzmenko can one of you please do a bit of follow up, here?

[stale]

Thank you for updating this issue. It is no longer marked as stale.