_ext_nodes unavailable on master

[jkur]

Setup:
Master: Debian with 2014.1.5+ds-1~bpo70+1
Minion: RHEL 6.5 with salt-minion-2014.1.4-1.el6.noarch

Almost all command regarding states fail.

Example:

salt-call state.show_highstate

[INFO ] Loading fresh modules for state activity
[INFO ] Fetching file from saltenv 'base', ** skipped ** latest already in cache 'salt://top.sls'
[INFO ] Fetching file from saltenv 'dev', ** skipped ** latest already in cache 'salt://top.sls'
Passed invalid arguments: 'bool' object is not iterable

On the master's log i get:
2014-06-18 15:19:46,896 [salt.master ][ERROR ] Received function _ext_nodes which is unavailable on the master, returning False

From 2014.4 to 2014.5 this function is missing from what i can see in the code.
Is it possible to make these Versions compatible? On RHEL the epel repo is always a bit behind, so i'm not able to install a newer version really...

Greetings

[jkur]

I could install 2014.5 from epel testing. So it's not a showstopper for me anymore. But i still think the master should handle that more reliable.

[basepi]

Hrm, that's no good! We don't guarantee that old masters will be able to talk to new minions (you should always upgrade your master first), but we definitely try to avoid it. Glad you got it solved, but we should look into this.

[mhworth]

I'm getting this error also, but for master and minion the same version (in fact, on the same machine):

ubuntu@master:~$ sudo salt-call state.show_highstate
[INFO    ] Loading fresh modules for state activity
[INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache 'salt://top.sls'
Passed invalid arguments: 'bool' object is not iterable
ubuntu@master:~$ sudo tail /var/log/salt/master
....
2014-06-20 01:38:47,116 [salt.master      ][ERROR   ] Received function _ext_nodes which is unavailable on the master, returning False
ubuntu@master:~$ salt-minion --version
salt-minion 2014.1.4
ubuntu@master:~$ salt-master --version
salt-master 2014.1.4

The salt environment was installed using

bootstrap-salt.sh -M

into an amazon EC2 instance running Ubuntu's vanilla 12.04 AMI. The master and minion are running on the same machine.

[mhworth]

FWIW, rebooting the master fixes this problem for me.

[sambanshee]

I also have a same issue.

[INFO ] Fetching file from saltenv 'dev', ** skipped ** latest already in cache 'salt://top.sls'
Passed invalid arguments: 'bool' object is not iterable

The problem here is that MASTER is of version 2014.5, but MINION 2014.4.1 partially fails to communicate with it.

Updating minion to 2014.5 from 2014.1 solved the issue.
However for example for Fedora 19 salt-minion 2014.5 is available only from testing repo, and I assume it is causing other issues as well.

[njones11]

I'm seeing this as well, also with a Fedora 19 minion. Any temporary workarounds until new Fedora packages are out?

[njones11]

@basepi Are you sure you wanted to close this? The issue that I'm seeing, and that the original poster was seeing is incompatibility between a 2014.1.5 master and a 2014.1.4 minion. I think this is a supported scenario, which implies to me that this may be a bug.

[mhworth]

This doesn't appear to be restricted to 2014.1.5 vs 2014.1.4 issue, as I
saw this same error when both master and minion were 2014.1.4.

On Fri, Jun 20, 2014 at 1:49 PM, Nick notifications@github.com wrote:

@basepi https://github.com/basepi Are you sure you wanted to close
this? The issue that I'm seeing, and that the original poster was seeing is
incompatibility between a 2014.1.5 master and a 2014.1.4 minion. I think
this is a supported scenario, which implies to me that this may be a bug.

—
Reply to this email directly or view it on GitHub
#13535 (comment).

[njones11]

@mhworth Unfortunately restarting the master doesn't fix it for me. I've restarted both master and minion, and still no luck.

[basepi]

Sounds like this may just be a bug in 2014.1.4. @sambanshee 2014.5 and 2014.1 are not real versions of salt. Can you clarify? Should be three parts, like 2014.1.1 or 2014.1.5

[njones11]

Temporary Fedora 19 workaround: yum update salt --enablerepo=updates-testing

[cro]

I have verified that this is indeed a problem between 2014.1.4 minions and 2014.1.5 master. It was caused by removing what we thought was some dead code (the old ext_nodes functionality that was replaced by the master_tops feature). It appears to only affect high state (or stuff that would access the master_tops subsystem)--I can run individual state files, execution modules, etc.

Here is a very short patch that can be applied against your master.py that can fix this until your minions are at 2014.1.5:

diff --git a/salt/master.py b/salt/master.py
index f5b4753..a869aff 100644
--- a/salt/master.py
+++ b/salt/master.py
@@ -950,6 +950,13 @@ class AESFuncs(object):
             return False
         return True

+    def _ext_nodes(self, load):
+        '''
+        Stub out until 2014.1.5 minion installed
+        '''
+        return {}
+
+
     def _master_opts(self, load):
         '''
         Return the master options to the minion

[sambanshee]

@basepi sorry for the typos. Of course, I meant 2014.1.4 and 2014.1.5 (and forgot the 1 in between)

[joehealy]

this patch has been applied in the debian and ubuntu (trusty and saucy) packages for 2014.1.5 versions salt_2014.1.5+ds-5*

[joehealy]

above comment means that for debian and ubuntu trusty and saucy, a salt-master installed from the packages should be able to interact with earlier salt-minions.

[cro]

Thanks @joehealy. Much appreciated.

[joehealy]

patched packages for ubuntu lucid (10.04) and precise(12.04) have just gone up to launchpad.

[cro]

@joehealy did I catch that you are cutting new releases due to #11523? The above patch is still somewhat broken for 2014.1.5. We've fixed it, but I would need to get a fresh patch to you for the deb pkgs. can you wait to cut it until I send to you?

[joehealy]

Waiting is easy - how long is it likely to be?

Cheers,

Joe

On Fri, Jun 27, 2014 at 2:28 PM, C. R. Oldham notifications@github.com
wrote:

@joehealy https://github.com/joehealy did I catch that you are cutting
new releases due to #11523 #11523?
The above patch is still somewhat broken for 2014.1.5. We've fixed it, but
I would need to get a fresh patch to you for the deb pkgs. can you wait to
cut it until I send to you?

—
Reply to this email directly or view it on GitHub
#13535 (comment).

[basepi]

Heh, if we're playing the waiting game, we may just want to wait until .6, which I'm hoping to cut later today.

[nshalman]

Did that patch make it into the 2014.1 branch? When will 2014.1.6 be cut?

[cro]

Yes, it made it. We plan to cut 2014.1.6 today, pending investigation of one more issue.