1.12.58: machine template generated doesn't match profile_base schema

[jouvin]

The HW machine schema generated by the broker doesn't match the quattor/type/hardware part of quattor/profile_base from template library 18.3.0, Identified issues are:

• room property of /hardware/rack is mandatory but it is not generated by the broker, even if it was defined when adding the rack. Instead broker adds a sysloc/room that seems to duplicate rack/id/room. Either the broker should add it to the generated HW machine template or the schema should be amended to make this property optional or remove it in favor of /sysloc/room. In addition, --room in aq add rack should be made mandatory if the current schema is kept.

• The broker adds /sysloc/country which is not part of the schema

In addition, the host data /system/archetype' generated by the broker seems completely unsynchronized with quattor/type/aquilon, also part of quattor/profile_base`:

• os_lifecycle is missing in the schema
• model (apparently used to store the OS version from the Aquilon OS object) is missing in the schema
• name is mandatory in the schema but not output by the broker (somewhat connected to 1.12.58: the broker should pass the archetype to the object template #88)
• os is a dict in the schema (with OS name and version) when it is a string in the hostdata

Also, the host data /system/personality' is suffering the same synchronisation problem as /system/archetype':

• systemgrn is required by the schema but not output by the broker

These are blocking issues for people relying on the template library. Workaround is easy to implement for experienced users but difficult to document for new users. May require a new template library once we decided what has to be changed to allow documenting a working configuration with the standard Aquilon and template library.

[ned21]

It looks like the external schema has drifted since hte schemas are not part of the open source push. Here's our latest personality schema:

type structure_personality = {
    "name"          : string
    "owner_eon_id"  : long
    "host_environment" : string with match(SELF, "^(dev|qa|uat|prod)$")
    "stage"         ? string
    "esp"           ? structure_espinfo
    "backups"       ? string
};

type structure_archetype = {
    "name"          : string # e.g. "aquilon"
    "os"            : string # e.g. "linux"
    "model"         : string # e.g. "7.1-x86_64"
    "filesystem-layout" : string with if_exists("archetype/filesystem-layouts/" + SELF) != ""
    "archlist"      : string[] # e.g. fs sysname list for model,
                               # "x86_64.linux.2.6.glibc.2.3", "amd64.linux.2.4.glibc.2.3", ...
    "os_lifecycle"  : string with match(SELF, "^(evaluation|pre_prod|early_prod|production|pre_decommission|inactive|withdrawn|decommissioned)$")
};

I think it might make more sense to file these drift issues as issues against the right TL repo not this one.

os is a dict in the schema (with OS name and version) when it is a string in the hostdata

This is a change that I remember @jrha making, presumably they have templates that populate this properly.

In addition, --room in aq add rack should be made mandatory if the current schema is kept.

It's not mandatory because there's an item, bunker, in the hierarchy that is a collection of racks within a room.

Rack is definitely added to the machine object:

$ aq cat --machine foo
structure template machine/americas/hz/hz864/foo;

"sysloc/continent" = "na";
"sysloc/country" = "us";
"sysloc/city" = "ss";
"sysloc/campus" = "ny";
"sysloc/building" = "hz";
"sysloc/bunker" = "nyb24.hz";
"rack/name" = "hz864";
"rack/row" = "bc";
"rack/column" = "009";
"location" = "hz.ss.na";

"nodename" = "foo";
include { "hardware/machine/hp/bl460cg8" };

"ram" = list(
  create("hardware/ram/generic",
    "size", 262144
  )
);
"cpu" = list(
  create("hardware/cpu/intel/e5-2660"),
  create("hardware/cpu/intel/e5-2660")
);
"harddisks/{sda}" = create("hardware/harddisk/generic/sata",
  "boot", true,
  "capacity", 373*1024,
  "interface", "sata"
);
"cards/nic/eth0" = create("hardware/nic/generic/generic_nic",
  "boot", true,
  "hwaddr", "xx:xx:xx:xx"
);
"cards/nic/eth1" = create("hardware/nic/generic/generic_nic",
  "hwaddr", "xx:xx:xx:xx"
);
"console/ilo" = nlist(
  "fqdn", "foo.ms.com",
  "hwaddr", "xx:xx:xx:xx"
);

I think we probably need @jrha to provide some insight on how they bridged the gap from the more recent QWG hardware/locaton schema from the older one that Aquilon works with.

[jouvin]

@ned21 please, could you share your version of the 3 templates I mentioned so that we understand if there is any change controversial or if we just need to merge your change in upstream templates. IMO, It'd be good if you could open pull request against template-library-core when you require a change to the schema, else this is a potential source of problem for Aquilon users... If you can attach your version to this issue this is great, else could you send them to me so that I can review the change and open the relevant pull request against template-library-core?

[ned21]

It'd be good if you could open pull request against template-library-core

I agree it would have been nice if we had done this in the past, and I think it would be reasonable to do so if there are future changes.

could you share your version of the 3 templates I mentioned

Which 3 sorry? I already did archetype and personality. In terms of templates, there has been some refactoring in the QWG version which moved lots of stuff around so it may be a case of tracking down structures rather than files.

[jrha]

In response to what we are doing for OS version information: it's all handled in the archetype's OS configs, for example this is os/sl/7x-x86_64/config.pan from the ral-tier1 archetype:

template os/sl/7x-x86_64/config;

include 'components/spma/config';

variable LOADPATH = append(SELF, 'ral-tier1/os/sl/7x-x86_64');

variable NODE_OS_VERSION_DEFAULT = 'sl7x-x86_64';
variable NODE_OS_VERSION = NODE_OS_VERSION_DEFAULT;
variable PKG_ARCH_DEFAULT = 'x86_64';

include 'os/sl/distribution';

prefix '/system/os';

'version/name' = '7x';
'version/major' = 7;

'architecture' = 'x86_64';

# Repositories related to base OS and quattor client (should be first)
include 'repository/config/os';
include 'os/version';


variable DEFAULT_FILESYSTEM_TYPE = 'xfs';

# Define xfs mount options for SL7x:
# Allow reading and writing
# Do not update inode access time
# Indicate to selinux that the filesystem is using xattrs for labels and that it supports label changes by setting the xattrs
# Enable "opportunistic" improvement to the way inline xattrs are stored on-disk
# Allow XFS to create inodes at any location in the filesystem, including those resulting in inode numbers occupying more than 32 bits of significance
# Forcibly turn off all quota accounting and enforcement within the fs
variable DEFAULT_MOUNTOPTS_XFS = 'rw,noatime,seclabel,attr2,inode64,noquota';

# We can use the generic el7 kickstart variant for sl7
variable AII_KS_OS_MAJOR_SPECIFIC_INCLUDE = 'quattor/aii/ks/variants/el7';

include 'quattor/client/config';

# Sendmail config to redirect mail for Tier1
include 'config/ral/tier1-sendmail';

# Set up sysctl
include 'site/config/sysctl-mmap';
include 'site/config/sysctl-large-network';

# Flag if reboot required after installing updates
include 'common/core/reboot-required/config';

prefix '/software/components/spma';
'process_obsoletes' = true;
'main_options/installonly_limit' = 2;

include 'components/systemd/legacy/chkconfig';

and os/sl/distribution.pan:

template os/sl/distribution;

prefix '/system/os';

'distribution/name' = 'sl';
'distribution/description' = 'Scientific Linux';
'distribution/family' = 'el';

[jouvin]

@ned21 yes, I'll open an issue/PR in the template-library-core repo... I also turned the description list into a list of task so that we can ensure that everything has been tackled. The 3 templates are those mentioned in the description from quattor/types.

[jouvin]

@ned21 @jrha my remark about /system/archetype is about what is filled by the broker. I think it must be consistent with the schema as it is assigned to the configuration path, not to a variable. @jrha approach is masking (working around) the problem but we should fix it!

[ned21]

I previously provided up to date schema for personality and archetype so I ticked those off. Here's the hardware structure we use:

type structure_hardware = {
    include structure_annotation
    "cpu"          ? structure_cpu[]
    "ram"          ? structure_ram[]
    "cards"        ? structure_cards
    "rack"         ? structure_rack
    # Obsolete field, use the appropriate "cards" sub-field instead!!
    "harddisks"    ? structure_raidport{} with check_harddisks(SELF)
    "chassis"      ? string
    "slot"         ? long
    "console"      ? structure_console
    "firmware"     ? string
    "livecd"       ? string
    "rescue"       ? string
    # should kernelargs be added to the hw template?
    "osargs"       ? structure_osargs
    "cmos"         ? string{}
    "support"      ? structure_hw_support
};
type structure_annotation = {
   "name"          ? string # "product name"
   "type"          ? string # "product type"
   "model"         ? string # "product model"
   "template_name" ? string with match(SELF, "^[a-z0-9_.-]+$") # "template name"
   "manufacturer"  ? string # "manufacturer name"
   "vendor"        ? string # "vendor name"
   "version"       ? string # "product version"
   "chipset"       ? string # "product chipset"
   "serialnumber"  ? string # "product serialnumber"
   "arch"          ? string # "product architecture" i386, amd64, m68k, ...
   "bus"           ? string # "bus of peripheral"
   "clock"         ? long   # "clock of peripheral"
   "lang"          ? string # "language of the product"
   "power"         ? long   # "power in watts"
   "location"      ? string # "location of the hardware"
   "nodename"      ? string # "unique hardware name of this device"
   "sysloc"        ? structure_sysloc
   "uri"           ? string
   "uuid"          ? type_uuid
   # ...
   # please do add your stuff here and enhance it by whatever you need!
};

type structure_sysloc = {
  "room"       ? string
  "bunker"     ? string
  "building"   ? string
  "city"       ? string
  "country"    ? string
  "region"     ? string
  "continent"  ? string
  "campus"     ? string
};

I suspect you will find structure_hardware is the QWG schema from ~8 years ago!

[jrha]

@stdweird and I spent quite a lot of effort re-factoring the schema and ensuring it was compatible with the (SCDB-style) QWG templates as well as aquilon.

I'm not sure what you are saying is missing. We are using the schema from template-library-core without modification.

[jrha]

Also see quattor/template-library-core#142 for the discussion around the OS structure.

[jouvin]

@jrha I don't know if your comment was for @ned21 or for me. I just pointed out, starting from scratch (it means Aquilon 1.12.58 + TL 18.3.0) that the schema and the behaviour of the broker are not consistent and that they need to be IMO.

[jrha]

I know we're running an older version of the broker, but it is populating rack/room...

# aq cat --machine system2176
structure template machine/gb/r89/r89rack105/system2176;

"sysloc/continent" = "europe";
"sysloc/city" = "harwell";
"sysloc/building" = "r89";
"sysloc/campus" = "harwell";
"rack/name" = "r89rack105";
"rack/row" = "1600";
"rack/column" = "2200";
"rack/room" = "lpd";
"location" = "r89.harwell.europe";
…

[jrha]

Ah, see 6b14139

[jouvin]

quattor/template-library-core#177 should bring a template library schema compatible with Aquilon, except for the OS where I kept the change to a dict() mentioned by @jrha . Rack room is populated by Aquilon if defined but you can also define a rack in a building without defining the room in which case it is not in the generated profile. In this PR, I decided that this property should become optional to reflect the Aquilon object.

[jouvin]

room property issue mentioned in the description has been fixed by the schema update, except the removal of the unused room property in structure_rack which is followed in quattor/template-library-core#182, as making the archetype name property mandatory.

For this reason, this items are marked done in this issue and this issue is closed.