hiera eyaml does not work on PE 3.7.2

[jeeslo]

Hello all,

Does anyone has eyaml working on the latest version of PE? if so, how?
I have upgraded from PE 3.3 to 3.7.2, and hiera eyaml stopped working. However, hiera (no encryption) keeps working as expected
I have also tried to set it up on a fresh installation of PE 3.7.2, with the same result...

The error I receive is when i run puppet agent -t:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find data item test:password in any Hiera data file and not default supplied at /etc/puppetlabs/puppet/environments/test/modules/accounts/manifests/init.pp:2 on node mynodexxxx.domain.com

in that init.pp, i have configured the following:

 class accounts{
            $credentials = hiera('test::password')
            ...

in mynodexxxx.domain.com.eyaml file i have the following:

test::password: >
    ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBADAFMAACAQEw
    DQYJKoZIhvcNAQEBBQAEggEAmAgX0ZCdwCtxQ7MkJ6FqGwhAS8mVDYAX24Pc
    lbwRZSQtQG/w0fYWiAC2KVeV6IDSIp8F/T48PJgAmK/Uq7c7bXhyPTB5mR52
    /tL9tlBhR+Wb+aaYjFmY+WLOs4kwx4k9XPHQuQmjX0wizDATaOR7E8Sojge4
    mPD3lLrpUkWT1l8Fn+5n1WI9oFDfz3GZGS5R/ITEqvMTm8t6GuWH3XZPHNyP
    j+KFQg7yBLsQYkA8WXYUtOzR+qKrJVDkKLnePRvlro5zpxhfZXZDYmUPpKHR
    PSwy4umbmo3d++E86Kn8Jy1uvyvT7jWW/CCbx3zYuIRexHsx323EitRBNiWZ
    4z9TBDA8BgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBAMZVsKJQz6ARcePmUs
    4XytgBC6w+37Un2OQJkBi6ZQ5Ml3]

in hiera.yaml i have the following:

 :backends:
  - eyaml
  - yaml
:hierarchy:
  - "node/%{::clientcert}"
  - "%{environment}"
  - common
:yaml:
  :datadir: /etc/puppetlabs/puppet/hieradata
:eyaml:
  :datadir: /etc/puppetlabs/puppet/hieradata
  :pkcs7_private_key: '/etc/puppetlabs/puppet/keys/private_key.pkcs7.pem'
  :pkcs7_public_key: '/etc/puppetlabs/puppet/keys/public_key.pkcs7.pem'
:logger: console

[elyscape]

Repaste your hiera.yaml file surrounded by ``` characters. Like this:

```
hiera.yaml
```

Anecdotally, I can tell you that hiera-eyaml works great for me on PE 3.7.2.

[sihil]

Which version are you using. We cut a new 2.0.4 this week although it included no backend changes I can think of.

[jeeslo]

I am glad to hear that you guys have eyaml working on PE 3.7.2 with no problems.

I am using the following versions:
hiera v1.3.4
eyaml v2.0.4

As far as i can see from the error message, it looks like whatever i define in .eyaml format is not picked up...however eyaml is installed and working (i can encrypt/decrypt strings without no issues).

I installed eyaml both times as follows:

$ sudo /opt/puppet/bin/gem install hiera-eyaml

is that way fully compatible with 3.7.x? should i use instead puppetserver gem install hiera-eyaml?.

any help or suggestion would be much appreciated.

[elyscape]

You'll need to do /opt/puppet/bin/puppetserver gem install hiera-eyaml, yeah. PE3.7 adds the puppetserver host.

[elyscape]

To elaborate some, the puppetserver system is a Java stack implementation of the Puppet master using JRuby. For security reasons, they locked down which folders it checks for gems and other libraries. Using puppetserver gem puts it in the correct folder. Using a regular gem install will install the executable if you want to use the eyaml commands while on the server, but it won't hook it into the master itself.

[jeeslo]

I receive the following error when i try to install it using puppetserver:

root@puppetserver01:/# /opt/puppet/bin/puppetserver gem install hiera-eyaml
ERROR:  Could not find a valid gem 'hiera-eyaml' (>= 0), here is why:
          Unable to download data from https://rubygems.org/ - certificate verify failed (https://rubygems.global.ssl.fastly.net/quick/Marshal.4.8/hiera-eyaml-2.0.4.gemspec.rz)
ERROR:  Possible alternatives: hiera-eyaml

[elyscape]

Is your Puppet master behind a firewall like Zscaler that does HTTPS filtering?

[jeeslo]

Yes, we are in a filtered network.
Can i download it and install it locally instead?

[elyscape]

Yes. This page has instructions for you. In the long term, I recommend submitting a ticket to your IT team to have them whitelist the Puppet master for bypassing the filter. If that sort of blanket filter bypass isn't possible, see if they can at least whitelist rubygems.global.ssl.fastly.net. That should be doable.

[jeeslo]

Hi elyscape,

The instructions listed involve to install the gem using:

$ sudo /opt/puppet/bin/gem install hiera-eyaml

instead of

$  sudo /opt/puppet/bin/puppetserver gem install hiera-eyaml

so i don't know whether that will work considering my scenario (the first command runs well in my environment, eyaml is installed and working, but puppet is not aware)
Is there any other way of doing that? I mean can i download the gem locally using wget and install it?
I can download the .rz file: https://rubygems.global.ssl.fastly.net/quick/Marshal.4.8/hiera-eyaml-2.0.4.gemspec.rz , but can i install it from that .rz?

[jeeslo]

well, in any case i have just followed the instructions suggested and after copying it from another puppet server and installing it, it keeps failing...
I am sure i am overlooking at something, what can i check to ensure that the eyaml is well configured?

[jeeslo]

in my attempt to resolve this issue, i have set up eyaml on a fresh new puppet master 3.7.3 (open source in this case), but i receive the same issue
eyaml can encrypt/decrypt with no issues, but it fails when its called from hiera

[elyscape]

Try changing these lines in your hiera.yaml file:

  :pkcs7_private_key: '/etc/puppetlabs/puppet/keys/private_key.pkcs7.pem'
  :pkcs7_public_key: '/etc/puppetlabs/puppet/keys/public_key.pkcs7.pem'

to:

  :pkcs7_private_key: /etc/puppetlabs/puppet/keys/private_key.pkcs7.pem
  :pkcs7_public_key: /etc/puppetlabs/puppet/keys/public_key.pkcs7.pem

That is to say, try removing the quotes.

[jeeslo]

I am afraid that removing the quotes did not work in neither of the two puppet servers I have (Enterprise and OpenSource)... I keep receiving the same error.
Is there any way I can check whether the eyaml backend is loaded?

[elyscape]

Basically, running /opt/puppet/bin/puppetserver gem list hiera-eyaml on a PE master or puppetserver gem list hiera-eyaml on an open-source master should give you output like this:

$ /opt/puppet/bin/puppetserver gem list hiera-eyaml

*** LOCAL GEMS ***

hiera-eyaml (2.0.4)

If it doesn't, then it's not installed properly. As for the site I Iinked giving commands along the lines of sudo gem whatever instead of sudo puppetserver gem whatever, just pretend each command has puppetserver or /opt/puppet/bin/puppetserver in front of gem, as appropriate for whichever version you're using. The site is for generic RubyGems, but it all applies to puppetserver's implementation.

[jeeslo]

Hi again,
the output in the Puppet Opensource (3.7.3) running CentOS 6.5 is the following:

$ /usr/bin/gem list hiera-eyaml

*** LOCAL GEMS ***

hiera-eyaml (2.0.4)

After upgrading Puppet-Enterprise (3.7.2) running Ubuntu 14.04, the following command did not give any output:

$ /opt/puppet/bin/puppetserver gem list hiera-eyaml

However running /opt/puppet/bin/gem reported the following:

$ /opt/puppet/bin/gem list hiera-eyaml

*** LOCAL GEMS ***

hiera-eyaml (2.0.3)

What i did after seeing this was to remove hiera-eyaml:

/opt/puppet/bin/gem uninstall hiera-eyaml

and install hiera-eyaml using puppetserver gem from a local folder which containts( hiera-eyaml-2.0.4.gem highline-1.6.21.gem trollop-2.0.gem)

$ /opt/puppet/bin/puppetserver gem list hiera-eyaml
*** LOCAL GEMS ***

hiera-eyaml (2.0.4)

Although that might seem the cause of the issue reported, after doing the steps listed above (upgrade, uninstall hiera-eyaml (2.0.3) and install (2.0.4) using puppetserver, i keep receiving the same issue...

[jeeslo]

One thing to point out:
In this attempt, I cannot see eyaml loaded in the PATH.
After searching for it, these are the results:

$ find / -name "eyaml"
/var/opt/lib/pe-puppet-server/jruby-gems/bin/eyaml
/var/opt/lib/pe-puppet-server/jruby-gems/gems/hiera-eyaml-2.0.4/bin/eyaml
/var/opt/lib/pe-puppet-server/jruby-gems/gems/hiera-eyaml-2.0.4/lib/hiera/backend/eyaml
/etc/eyaml

When i try to run it:

/var/opt/lib/pe-puppet-server/jruby-gems/gems/hiera-eyaml-2.0.4/bin/eyaml

I receive the following error:

/usr/bin/env: ruby: No such file or directory

I have added the ruby path:

export PATH=$PATH:/opt/puppet/bin/

And now i receive the following errors:

/opt/puppet/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- hiera/backend/eyaml/CLI (LoadError)
        from /opt/puppet/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
        from /var/opt/lib/pe-puppet-server/jruby-gems/gems/hiera-eyaml-2.0.4/bin/eyaml:4:in `<main>'

[elyscape]

This is normal. When you install hiera-eyaml into puppetserver, it won't add it to your PATH, but it will make it available to the Puppet master. It should work as a backend now. If you also want it available on the command line, do a regular gem install as well.

[jeeslo]

I would like to say something different, but still does not work...
I don't know what else i can check..

[elyscape]

So does it currently not work as a backend for hiera on puppetserver in addition to not working on the command line?

[jeeslo]

I have to restore the VM, but it definetely was not working neither by command line nor from hiera.
So i have not been able to resolve it yet..

[elyscape]

Try running /opt/puppet/bin/puppetserver gem install hiera-eyaml and then running service pe-puppetserver restart. On a non-PE system, those commands would be puppetserver gem install hiera-eyaml and service puppetserver restart.

[ipcrm]

In case someone is still struggling, to do this behind a proxy I had to use the following:

/opt/puppet/bin/puppetserver gem install -p "http://<user>:<pass>@<proxy>:<port>" hiera-eyaml

[malnick]

Who here has actually gotten eyaml to reliably run on 3.7x?

[elyscape]

I have.

[malnick]

Awesome, I'm going through the process of trying this again. I had to drop back to 3.3 as I ran into some blockers on this a few months ago and didn't have the time to actually figure it out. I think it was a faulty gem install on the PM, so I'm trying again today.

[malnick]

Out of curiosity, can I actually execute eyaml encrypt commands on the PM (running PE not OS) or is it only available to the PM process?

[elyscape]

To be able to use eyaml on the command line, you'll need to also install the gem using the regular gem command. If you only install it via the puppetserver gem command, it'll only be available to the Puppet Server stack.

[malnick]

That's what I thought. On my last go around I was really confused about that.

[malnick]

Did you see something like this:

ERROR:  Could not find a valid gem 'hiera-eyaml' (>= 0), here is why:
Unable to download data from https://rubygems.org/ - certificate verify failed (https://rubygems.global.ssl.fastly.net/quick/Marshal.4.8/hiera-eyaml-2.0.7.gemspec.rz)
ERROR:  Possible alternatives: hiera-eyaml

[malnick]

^^ with puppetserver gem install

I've seen this issue on windows but not ubuntu before.

[elyscape]

Are you behind a firewall that does SSL interception/inspection?

[malnick]

I'm in aws but 443 is open. My quick solution which is probably bad was:

 ./puppetserver gem sources -r https://rubygems.org/
 ./puppetserver gem sources -a http://rubygems.org/

Then the install worked fine.

[malnick]

Notice: hiera(): Cannot load backend eyaml: cannot load such file -- hiera/backend/eyaml_backend

[malnick]

^^ Installed with puppetserver, this was the point that I got to before and gave up.

[malnick]

---
:backends:
  - eyaml
  - yaml
:hierarchy:
  - "roles/%{role}"
  - global
  - encrypted

:yaml:
  :datadir: /etc/puppetlabs/puppet/environments/%{environment}/hieradata

:eyaml:
  :datadir: /etc/puppetlabs/puppet/environments/%{environment}/hieradata
  :pkcs7_private_key: /etc/puppetlabs/puppet/ssl/keys/private_key.pkcs7.pem
  :pkcs7_public_key: /etc/puppetlabs/puppet/ssl/keys/public_key.pkcs7.pem
  :extension: 'yaml'

[elyscape]

You made sure to have the keys listed in /etc/puppetlabs/puppet/hiera.yaml, yes?

[malnick]

This hiera config worked fine on my 3.3 master

[elyscape]

What command are you running that produces this error?

[malnick]

puppet apply -e "notice(hiera('r10k_git_key_pub'))"

[malnick]

^^ An encrypted key from encrypted.yaml. Again, a key that decrypts fine with this exact config on my 3.3 master.

[elyscape]

For puppet apply, you'll need to have installed the gem via /opt/puppet/bin/gem install.

[malnick]

So. Many. Gems.

Ok I'll try that.

[malnick]

Oh shit! It worked! Like a boss, thanks.

[gapotts]

As have I.

From: Eli Young <notifications@github.commailto:notifications@github.com>
Reply-To: TomPoulton/hiera-eyaml <reply@reply.github.commailto:reply@reply.github.com>
Date: Sunday, April 5, 2015 at 3:08 PM
To: TomPoulton/hiera-eyaml <hiera-eyaml@noreply.github.commailto:hiera-eyaml@noreply.github.com>
Subject: Re: [hiera-eyaml] hiera eyaml does not work on PE 3.7.2 (#126)

I have.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/126#issuecomment-89862909.

This communication is Confidential Information. By using this message and attachments you implicitly consent to terms and conditions set forth at http://www.taos.com/email_disclaimer. If you do not consent or received this message in error, please destroy it.

[ltutar]

Following the steps described by @elyscape helped me get it working on PE2015.3.
Thanx.

[elyscape]

Just to consolidate everything:

• To get eyaml working on a Puppet Master that uses Puppet Server:
  /opt/puppet/bin/puppetserver gem install hiera-eyaml
• To get eyaml working with puppet apply:
  /opt/puppet/bin/gem install hiera-eyaml
• To get the eyaml binary on the command line:
  gem install hiera-eyaml

[jkumar19]

I am facing this same error when I run "puppet agent -t" on agent node. It is working fine with "puppet apply" on master.

[elyscape]

@jkumar19 When you run puppet agent -t on the node, are you running it against the master or standalone?

[jkumar19]

I am running against puppet master. I am not using standalone setup.

[Dan33l]

it can be reopened if it's reproducible on the latest Puppet 4 or newer