Report ENOEXEC error as "command failed to execute" #54000
Description
Description of Issue
I set up salt-proxy to run fx2/racadm. However upon startup, salt-proxy logged error "command not found".
This was very confusing, because I was able to run racadm from the command line, using the same $PATH.
After some digging, I discovered racadm was a symlink to an executable file, but the executable was missing a #!/usr/bin/env bash shebang. Bash followed the symlinks and ran the executable, but salt-proxy would not. Once I added the shebang, salt-proxy was able to run racadm.
Proposed solutions
To reduce confusion for future users, perhaps it would make sense to either:
- Report "command failed to execute" as opposed to "command not found"
- Alternately, allow salt-proxy to execute in this case, since bash is able to do so
Setup
# grep -v ^# /etc/salt/proxy | grep -v ^$
master: my-master
pillar_roots:
base:
- /etc/salt/pillar
# cat /etc/salt/pillar/top.sls
base:
my-host-idrac:
- my-host-idrac
# cat /etc/salt/pillar/my-host-idrac.sls
proxy:
proxytype: fx2
host: my-host-idrac
admin_username: my-admin
passwords:
- my-pass
Steps to Reproduce Issue
# strace -f -o /tmp/proxy.out salt-proxy --proxyid my-host-idrac -l debug
Trimmed output other than stack trace:
[CRITICAL] Failed to load grains defined in grain file fx2.fx2 in function <function fx2 at 0x7f7efcef7398>, error:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 798, in grains
ret = funcs[key](**kwargs)
File "/usr/lib/python2.7/dist-packages/salt/grains/fx2.py", line 87, in fx2
return _grains()
File "/usr/lib/python2.7/dist-packages/salt/grains/fx2.py", line 69, in _grains
(username, password) = _find_credentials()
File "/usr/lib/python2.7/dist-packages/salt/grains/fx2.py", line 47, in _find_credentials
admin_password=pwd)
File "/usr/lib/python2.7/dist-packages/salt/modules/dracr.py", line 1097, in get_chassis_name
admin_password=admin_password)
File "/usr/lib/python2.7/dist-packages/salt/modules/dracr.py", line 1394, in bare_rac_cmd
admin_password=admin_password)
File "/usr/lib/python2.7/dist-packages/salt/modules/dracr.py", line 128, in __execute_ret
output_loglevel='quiet')
File "/usr/lib/python2.7/dist-packages/salt/modules/cmdmod.py", line 2069, in run_all
**kwargs)
File "/usr/lib/python2.7/dist-packages/salt/modules/cmdmod.py", line 669, in _run
raise CommandExecutionError(msg)
CommandExecutionError: Unable to run command 'REDACTED' with the context '{u'timeout': None, u'with_communicate': True, u'shell': False, u'bg': False, u'stderr': -1, u'env': {'LC_NUMERIC': 'C', 'LC_CTYPE': 'C', 'LESSOPEN': '| /usr/bin/lesspipe %s', 'LOGNAME': 'root', 'USER': 'root', 'HOME': '/home/greenmoss', 'LC_PAPER': 'C', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin', 'DISPLAY': 'localhost:11.0', 'LANG': 'en_US.UTF-8', 'TERM': 'screen', 'SHELL': '/bin/bash', 'LANGUAGE': 'C', 'LC_MEASUREMENT': 'C', 'SUDO_USER': 'greenmoss', 'LC_MONETARY': 'C', 'USERNAME': 'root', 'LC_IDENTIFICATION': 'C', 'LC_ADDRESS': 'C', 'SUDO_UID': '12345', '_': '/usr/bin/strace', 'SUDO_COMMAND': '/bin/bash', 'SUDO_GID': '1234', 'LESSCLOSE': '/usr/bin/lesspipe %s %s', 'LC_MESSAGES': 'C', 'OLDPWD': '/etc/salt', 'LC_TELEPHONE': 'C', 'SHLVL': '1', 'PWD': '/opt/local/bin', 'LC_NAME': 'C', 'MAIL': '/var/mail/root', 'LC_TIME': 'C', 'LC_COLLATE': 'C'}, u'stdout': -1, u'close_fds': True, u'stdin': None, u'cwd': u'/home/greenmoss'}', reason: command not found
trimmed exit messages, now looking at relevant section from strace:
32535 execve("/usr/local/sbin/racadm", ["racadm", "-r", "my-host-idrac", "-u", "my-admin", "-p", "my-pass", "getchassisname"], [/* 34 vars */]) = -1 ENOENT (No such file or directory)
32535 execve("/usr/local/bin/racadm", ["racadm", "-r", "my-host-idrac", "-u", "my-admin", "-p", "my-pass", "getchassisname"], [/* 34 vars */]) = -1 ENOENT (No such file or directory)
32535 execve("/usr/sbin/racadm", ["racadm", "-r", "my-host-idrac", "-u", "my-admin", "-p", "my-pass", "getchassisname"], [/* 34 vars */]) = -1 ENOENT (No such file or directory)
32535 execve("/usr/bin/racadm", ["racadm", "-r", "my-host-idrac", "-u", "my-admin", "-p", "my-pass", "getchassisname"], [/* 34 vars */]) = -1 ENOENT (No such file or directory)
32535 execve("/sbin/racadm", ["racadm", "-r", "my-host-idrac", "-u", "my-admin", "-p", "my-pass", "getchassisname"], [/* 34 vars */]) = -1 ENOENT (No such file or directory)
32535 execve("/bin/racadm", ["racadm", "-r", "my-host-idrac", "-u", "my-admin", "-p", "my-pass", "getchassisname"], [/* 34 vars */]) = -1 ENOEXEC (Exec format error)
32535 execve("/snap/bin/racadm", ["racadm", "-r", "my-host-idrac", "-u", "my-admin", "-p", "my-pass", "getchassisname"], [/* 34 vars */]) = -1 ENOENT (No such file or directory)
Note the ENOEXEC in the middle there, even though I can run it from shell:
# racadm help
help [subcommand] -- display usage summary for a subcommand
clearasrscreen -- clear the last ASR (crash) screen
(trimmed)
Now the weird chain of links, etc:
# which racadm
/bin/racadm
# ls -l /bin/racadm
lrwxrwxrwx 1 root root 30 Nov 23 2016 /bin/racadm -> /opt/dell/srvadmin/sbin/racadm
# ls -l /opt/dell/srvadmin/sbin/racadm
lrwxrwxrwx 1 root root 21 Nov 23 2016 /opt/dell/srvadmin/sbin/racadm -> racadm-wrapper-idrac7
# cat /opt/dell/srvadmin/sbin/racadm-wrapper-idrac7
#directly invoke idracadm7 - it will check for generation and invoke required binary
exec /opt/dell/srvadmin/bin/idracadm7 "$@"
Fix: edit /opt/dell/srvadmin/sbin/racadm-wrapper-idrac7, add shebang:
#!/usr/bin/env bash
#directly invoke idracadm7 - it will check for generation and invoke required binary
exec /opt/dell/srvadmin/bin/idracadm7 "$@"
Versions Report
Salt Version:
Salt: 2019.2.0
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 1.5
docker-py: Not Installed
gitdb: 0.5.4
gitpython: 0.3.2 RC1
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: 0.21.1
Mako: 0.9.1
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: 1.2.3
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.6 (default, Oct 26 2016, 20:30:19)
python-gnupg: Not Installed
PyYAML: 3.10
PyZMQ: 14.0.1
RAET: Not Installed
smmap: 0.8.2
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.0.4
System Versions:
dist: Ubuntu 14.04 trusty
locale: UTF-8
machine: x86_64
release: 4.4.0-31-generic
system: Linux
version: Ubuntu 14.04 trusty