Start moving away from search queries to static config items for cookbooks #947

vuntz · 2017-01-03T12:05:59Z

The static config items are actually data bag items (for now).

The main benefits are:

performance improvement (as a search query can be expensive)
going in the direction of not depending on centralized chef-server
in some cases, simplification of the cookbooks

The rake task will be used from barclamp_install.rb so that the config items get regenerated when the code is updated (so that there's no need to reapply a barclamp to get an up-to-date config item with the new fields).

~~Marking as WIP as I'd like to gather feedback; but this seems to work quite well.~~

houndci-bot · 2017-01-03T12:06:10Z

crowbar_framework/lib/tasks/crowbar.rake

+
+    barclamps.each do |barclamp|
+      begin
+        cls = eval("#{barclamp.camelize}Service")


Lint/Eval: The use of eval is a serious security risk.

houndci-bot · 2017-01-03T12:06:10Z

crowbar_framework/lib/tasks/crowbar.rake

+
+    barclamps.each do |barclamp|
+      begin
+        cls = eval("#{barclamp.camelize}Service")


Lint/Eval: The use of eval is a serious security risk.

houndci-bot · 2017-01-03T12:06:10Z

crowbar_framework/lib/crowbar/data_bag_config.rb

+
+        Chef::DataBagItem.load(data_bag_name, group)
+      rescue Net::HTTPServerException
+        db = begin


Lint/UselessAssignment: Useless assignment to variable - db. (https://github.com/bbatsov/ruby-style-guide#underscore-unused-vars)

houndci-bot · 2017-01-03T12:06:10Z

crowbar_framework/lib/crowbar/data_bag_config.rb

+
+        Chef::DataBagItem.load(data_bag_name, group)
+      rescue Net::HTTPServerException
+        db = begin


Lint/UselessAssignment: Useless assignment to variable - db. (https://github.com/bbatsov/ruby-style-guide#underscore-unused-vars)

houndci-bot · 2017-01-03T12:06:10Z

chef/cookbooks/uwsgi/providers/default.rb


-  execute "pip install --index-url http://#{proxy_addr}:#{proxy_port}/files/pip_cache/simple/ uwsgi" do
+  execute "pip install --index-url #{provisioner_config["root_url"]}/files/pip_cache/simple/ uwsgi" do


Metrics/LineLength: Line is too long. [102/100] (https://github.com/SUSE/style-guides/blob/master/Ruby.md#metricslinelength)

houndci-bot · 2017-01-03T12:06:10Z

chef/cookbooks/uwsgi/providers/default.rb


-  execute "pip install --index-url http://#{proxy_addr}:#{proxy_port}/files/pip_cache/simple/ uwsgi" do
+  execute "pip install --index-url #{provisioner_config["root_url"]}/files/pip_cache/simple/ uwsgi" do


Metrics/LineLength: Line is too long. [102/100] (https://github.com/SUSE/style-guides/blob/master/Ruby.md#metricslinelength)

houndci-bot · 2017-01-03T12:06:10Z

chef/cookbooks/ntp/recipes/default.rb

  template "/etc/ntp.conf" do
    owner "root"
    group "root"
    mode 0644
    source "ntp.conf.erb"
    variables(ntp_servers: ntp_servers,
-            admin_interface: admin_interface,
+            admin_interface: local_admin_address,


Style/AlignHash: Align the elements of a hash literal if they span more than one line.

houndci-bot · 2017-01-03T12:06:10Z

chef/cookbooks/ntp/recipes/default.rb

  template "/etc/ntp.conf" do
    owner "root"
    group "root"
    mode 0644
    source "ntp.conf.erb"
    variables(ntp_servers: ntp_servers,
-            admin_interface: admin_interface,
+            admin_interface: local_admin_address,


Style/AlignHash: Align the elements of a hash literal if they span more than one line.

houndci-bot · 2017-01-03T12:06:10Z

chef/cookbooks/network/recipes/default.rb

+provisioner_address = provisioner_config["server"]
+
+if provisioner_address
+  Chef::Log.info("Checking we can ping #{provisioner_address}; " +


Style/LineEndConcatenation: Use \ instead of + or << to concatenate those strings.

houndci-bot · 2017-01-03T12:06:11Z

chef/cookbooks/network/recipes/default.rb

+provisioner_address = provisioner_config["server"]
+
+if provisioner_address
+  Chef::Log.info("Checking we can ping #{provisioner_address}; " +


Style/LineEndConcatenation: Use \ instead of + or << to concatenate those strings.

houndci-bot · 2017-01-03T12:06:11Z

chef/cookbooks/barclamp/libraries/barclamp_library.rb

@@ -377,5 +377,60 @@ def self.size_to_bytes(s)
        end
      end
    end
+
+    class Config
+      def self.node=(node)


Style/TrivialAccessors: Use attr_writer to define trivial writer methods. (https://github.com/bbatsov/ruby-style-guide#attr_family)

houndci-bot · 2017-01-03T12:06:11Z

chef/cookbooks/barclamp/libraries/barclamp_library.rb

@@ -377,5 +377,60 @@ def self.size_to_bytes(s)
        end
      end
    end
+
+    class Config
+      def self.node=(node)


Style/TrivialAccessors: Use attr_writer to define trivial writer methods. (https://github.com/bbatsov/ruby-style-guide#attr_family)

vuntz · 2017-01-09T16:47:59Z

IMHO, this is ready for reviews (& merging, if people are fine with this).

vuntz · 2017-01-09T16:48:34Z

Oh, we'll want crowbar/crowbar#2302 at the same time.

houndci-bot · 2017-01-11T17:38:19Z

chef/cookbooks/uwsgi/providers/default.rb

-  proxy_addr = provisioner[:fqdn]
-  proxy_port = provisioner[:provisioner][:web_port]
+  provisioner_config = BarclampLibrary::Barclamp::Config.load("core", "provisioner")
+  index_url = "#{provisioner_config["root_url"]}/files/pip_cache/simple/"


Style/StringLiteralsInInterpolation: Prefer single-quoted strings inside interpolations.

houndci-bot · 2017-01-11T17:38:19Z

chef/cookbooks/crowbar/recipes/prepare-upgrade-scripts.rb

 old_install_url = "#{web_path}/install"

-web_path = "#{provisioner_web}/#{node[:target_platform]}/#{arch}"
+web_path = "#{provisioner_config["root_url"]}/#{node[:target_platform]}/#{arch}"


Style/StringLiteralsInInterpolation: Prefer single-quoted strings inside interpolations.

houndci-bot · 2017-01-11T17:38:19Z

chef/cookbooks/crowbar/recipes/prepare-upgrade-scripts.rb

-provisioner_web = "http://#{admin_ip}:#{web_port}"
-
-web_path = "#{provisioner_web}/#{node[:platform]}-#{node[:platform_version]}/#{arch}"
+web_path = "#{provisioner_config["root_url"]}/#{node[:platform]}-#{node[:platform_version]}/#{arch}"


Style/StringLiteralsInInterpolation: Prefer single-quoted strings inside interpolations.

The goal of this is to allow barclamps to export the relevant config bits to data bag items, so that nodes that are either not part of this proposal but need config from that barclamp or nodes that require info about other nodes in this proposal, can simply load the data bag item. We also cache the config to avoid loading it multiple times per run. This will help reduce the amount of search queries that are done from the clients to the chef server.

There are more uses of search queries for the provisioner-server node, but let's ignore that for now.

This will be used when we need to load the config from the rails app (to avoid looking at proposals).

When no instance is specified and we can't detect which one to use, then try with "default", and if that doesn't work, find the first instance that has an applied proposal for this barclamp. This makes it possible to do something like Config.load("openstack", "nova") without having to care what's the instance of the applied nova proposal -- which is okay because there's only one applied proposal for that barclamp.

On installation, the data bag configuration is created before the admin server gets allocated an IP address. So we need to deal with the case where the node may not have an admin network in the code. This also means we will need to regenerate the data bag configuration at some point; this will be done in the install script, after we move the admin server to hardware-installing.

dirkmueller

can you fix the houndci issues ? I'm in general happy about the change otherwise.

This allows us to use attr_accessor.

vuntz · 2017-01-12T13:57:47Z

@dirkmueller Added two commits to fix the remaining hound comments.

vuntz added the wip label Jan 3, 2017

vuntz assigned aspiers, matelakat and rsalevsky Jan 3, 2017

houndci-bot reviewed Jan 3, 2017

View reviewed changes

dirkmueller added this to the Cloud 7 Update2 milestone Jan 3, 2017

vuntz mentioned this pull request Jan 4, 2017

Start moving away from search queries to static config items for cookbooks crowbar/crowbar-openstack#718

Closed

vuntz force-pushed the databagconfig branch from 5d6e126 to 1acb91a Compare January 4, 2017 15:06

vuntz mentioned this pull request Jan 5, 2017

crowbar: Add rake task to regenerate data bag configs #954

Merged

vuntz force-pushed the databagconfig branch 5 times, most recently from 8245f17 to 0897ff5 Compare January 9, 2017 13:19

vuntz removed the wip label Jan 9, 2017

vuntz force-pushed the databagconfig branch from 0897ff5 to 3505309 Compare January 11, 2017 17:38

houndci-bot reviewed Jan 11, 2017

View reviewed changes

vuntz added 9 commits January 12, 2017 09:34

dns: Use new data bag item config infrastructure

5845ff7

ntp: Use new data bag item config infrastructure

5773887

logging: Use new data bag item config infrastructure

f7e8ff5

provisioner: Start using new data bag item config infrastructure

8d9265f

There are more uses of search queries for the provisioner-server node, but let's ignore that for now.

crowbar: Add Crowbar::DataBagConfig.load() method

be281af

This will be used when we need to load the config from the rails app (to avoid looking at proposals).

Make hound happy

2e65b51

vuntz force-pushed the databagconfig branch from 3505309 to 3963959 Compare January 12, 2017 08:34

dirkmueller modified the milestones: Cloud 7 GM, Cloud 7 Update2 Jan 12, 2017

dirkmueller self-requested a review January 12, 2017 09:26

dirkmueller approved these changes Jan 12, 2017

View reviewed changes

dirkmueller requested changes Jan 12, 2017

View reviewed changes

vuntz added 2 commits January 12, 2017 14:53

barclamp: Use an Eigenclass for Barclamp::Config

e38e302

This allows us to use attr_accessor.

Fix Style/StringLiteralsInInterpolation hound comments

0e5724a

dirkmueller approved these changes Jan 12, 2017

View reviewed changes

rsalevsky approved these changes Jan 16, 2017

View reviewed changes

dirkmueller merged commit b25ac67 into crowbar:master Jan 16, 2017

aspiers mentioned this pull request Feb 8, 2017

Make sure we're running with correct time on installation #568

Open

vuntz mentioned this pull request Mar 23, 2017

neutron recipes on compute nodes should not look at attributes from neutron-server nodes sap-oc/crowbar-openstack#11

Open

5 tasks

matelakat mentioned this pull request Mar 27, 2017

Start moving away from search queries to static config items for cookbooks sap-oc/crowbar-core#22

Open

Itxaka mentioned this pull request Aug 16, 2017

Fix databag approach crowbar/crowbar-openstack#1143

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Start moving away from search queries to static config items for cookbooks #947

Start moving away from search queries to static config items for cookbooks #947

vuntz commented Jan 3, 2017 •

edited

Loading

houndci-bot Jan 3, 2017

houndci-bot Jan 3, 2017

houndci-bot Jan 3, 2017

houndci-bot Jan 3, 2017

houndci-bot Jan 3, 2017

houndci-bot Jan 3, 2017

houndci-bot Jan 3, 2017

houndci-bot Jan 3, 2017

houndci-bot Jan 3, 2017

houndci-bot Jan 3, 2017

houndci-bot Jan 3, 2017

houndci-bot Jan 3, 2017

vuntz commented Jan 9, 2017

vuntz commented Jan 9, 2017

houndci-bot Jan 11, 2017

houndci-bot Jan 11, 2017

houndci-bot Jan 11, 2017

dirkmueller left a comment

vuntz commented Jan 12, 2017


		execute "pip install --index-url http://#{proxy_addr}:#{proxy_port}/files/pip_cache/simple/ uwsgi" do
		execute "pip install --index-url #{provisioner_config["root_url"]}/files/pip_cache/simple/ uwsgi" do

Start moving away from search queries to static config items for cookbooks #947

Start moving away from search queries to static config items for cookbooks #947

Conversation

vuntz commented Jan 3, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vuntz commented Jan 9, 2017

vuntz commented Jan 9, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dirkmueller left a comment

Choose a reason for hiding this comment

vuntz commented Jan 12, 2017

vuntz commented Jan 3, 2017 •

edited

Loading