Cannot use runTest if a different specialArgs is required per node

[yajo]

Describe the bug

Using nixos-lib.runTest doesn't allow to pass different node.specialArgs per node when using various nodes.

Steps To Reproduce

N/A (feature request)

Expected behavior

Since it supports using various nodes, runTest should also support passing different specialArgs per node.

Screenshots

N/A

Additional context

I deploy machines using terraform. Those deployments output a lock file that I use to generate nixosConfigurations. Each machine is different. I want to test that using runTest. These machines produce a k8s cluster, so they need to be tested together.

The machines get an inventory argument that allows them to know about themselves and about each other. As explained in the forum, using _module.args produces an infinite recursion, so the only valid solution is to pass the inventory as specialArgs.

If I cannot pass a specialArgs per node, nodes cannot know about themselves and about each other, and thus the cluster cannot be tested with VMs as closely to production configuration as I'd want to.

This is where you can define multiple nodes for a test:

nixpkgs/nixos/lib/testing/nodes.nix

Lines 41 to 53 in 7d0ed7f

	nodes = mkOption {
	type = types.lazyAttrsOf config.node.type;
	visible = "shallow";
	description = mdDoc ''
	An attribute set of NixOS configuration modules.
	The configurations are augmented by the [`defaults`](#test-opt-defaults) option.
	They are assigned network addresses according to the `nixos/lib/testing/network.nix` module.
	A few special options are available, that aren't in a plain NixOS configuration. See [Configuring the nodes](#sec-nixos-test-nodes)
	'';
	};

This is where you can define only one single specialArgs shared across all nodes:

nixpkgs/nixos/lib/testing/nodes.nix

Lines 71 to 79 in 7d0ed7f

	node.specialArgs = mkOption {
	type = types.lazyAttrsOf types.raw;
	default = { };
	description = mdDoc ''
	An attribute set of arbitrary values that will be made available as module arguments during the resolution of module `imports`.
	Note that it is not possible to override these from within the NixOS configurations. If you argument is not relevant to `imports`, consider setting {option}`defaults._module.args.<name>` instead.
	'';
	};

To fix the problem, a possible solution would be to support the case when node.specialArgs is a function. In that case, the function could be called with the machine name as the 1st argument, and the specialArgs would be the attrset obtained from that function. Example:

{
  inventory,
  specialArgs,
  lib,
  nixos-lib,
  pkgs,
  self,
  system,
}: let
  mkHost = hostName: hostData: {config, ...}: {
    imports = with self.outputs.nixosModules; [default interactiveFlake];
  };
  getSpecialArgs = node: specialArgs.${node};
in
  nixos-lib.runTest {
    name = "k8s-cluster";
    nodes = builtins.mapAttrs mkHost inventory;
    node.specialArgs = getSpecialArgs; # This is the important part: it's a function
    hostPkgs = pkgs;

    testScript = ''
      assert False, "failing works"
    '';
  }

Notify maintainers

@roberth

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.10-200.fc37.x86_64, Fedora Linux, 37.20230214.0 (Silverblue), nobuild`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.12.0`
 - channels(yajo): `"nixpkgs"`
 - nixpkgs: `/var/home/yajo/.nix-defexpr/channels/nixpkgs`

@moduon MT-1075

[roberth]

We could get the name from a new attrsOf, and then have nodes be like

  nodes = mkOption {
    type = dependentAttrsOf (name: submoduleWith { specialArgs = config.node.makeSpecialArgs name; /*...*/ });

    dependentAttrsOf = elemTypeFn:
    let
      # Only suitable for documentation
      staticElemType = elemTypeFn "<name>";
    in mkOptionType rec {
      name = "lazyAttrsOf";
      description = "lazy attribute set of ${optionDescriptionPhrase (class: class == "noun" || class == "composite") staticElemType}";
      descriptionClass = "composite";
      check = isAttrs;
      merge = loc: defs:
        zipAttrsWith (name: defs:
          let merged = mergeDefinitions (loc ++ [name]) (elemTypeFn name) defs;
          # mergedValue will trigger an appropriate error when accessed
          in merged.optionalValue.value or (elemTypeFn name).emptyValue.value or merged.mergedValue
        )
        # Push down position info.
        (map (def: mapAttrs (n: v: { inherit (def) file; value = v; }) def.value) defs);
      emptyValue = { value = {}; };
      getSubOptions = prefix: staticElemType.getSubOptions (prefix ++ ["<name>"]);
      getSubModules = staticElemType.getSubModules;
      substSubModules = m: dependentAttrsOf (name: (elemTypeFn name).substSubModules m);
      functor = (defaultFunctor name) // { wrapped = elemTypeFn; };
      nestedTypes.elemType = staticElemType;
    };

[roberth]

@infinisil What do you think of dependentAttrsOf above? I think it solves the problem quite nicely while keeping the syntax for the general case simple; I don't think nodes.<name>.{settings,specialArgs} would be an improvement, or a viable migration, or a nodes syntax that we should encourage in other tools.

I think it might also solve the problem with name in nested attrsOf, although that's a bit of an antipattern (dicts in dicts should be dicts in records in dicts for extensibility / attrsOf in attrsOf should be attrsOf in submodule in attrsOf for extensibility)

[infinisil]

I am not convinced we need to invent a new solution to this problem. Looking at the original post using _module.args (you could also create your own typed option), how about instead of

pinnedConfigs.${host.provider_name}.${host.deployment_version}
// {
  boot.loader.grub.device = disk;
}

Use mkIf:

{
  fileSystems."/" = lib.mkIf is_this_the_right_host_and_version {
    device = disk;
    fsType = "ext4";
  };
  boot.loader.grub.device = disk;
}

[roberth]

While the element type is significantly different for this variation of attrsOf, we're gathering multiple parameters by now.
Consider also restricting which attrNames are valid. We should probably switch to named arguments.

• module system: types.attrsWith { nameType, valueType, ... }? #193311

Axes of variation

• support mkIf (attrsOf) or self-references (lazyAttrsOf)
• restrict which attr names are valid
• allow the element type to be a function of the attr name or not

The latter two are fun; specifying a type should allow the type to convert the string attrName to a better type, such as bool or int, using coercedTo or something a bit more specialized. We may also consider extending the types to have parsers.