nixosTest matrix (prev also modularity, option docs)

[roberth]

Description of changes

Use the module system to replace the NixOS test framework's custom glue.

• NixOS tests can now use imports and other module system features.
• Type checking for the test arguments.
• Tests can now define overrides for interactive use.
• Network-wide test setup can now be reusable; example:
  • A setup with all VMs configured to use a DNS server
  • Split long, slow tests into multiple tests that import a
    common module that has most of the setup.
• We now have a module for test matrices / parameterized tests. We can stop reinventing the wheel, which we've done in many ways.
  • Generic support for version matrix + package passthru pattern (TBD)
• (TBD) "generated" options reference docs
• Implementation cleanup: aspects that had to be wired through all the glue are now in their own files.
  • testing/interactive.nix: just a couple of lines that can be ignored by the other modules, as opposed to wiring it through what is now testing/run.nix.
  • testing/network.nix, etc
• For supporting containerized tests, the flexibility of the module system will be useful.

All in all, this quite significantly collapses the number of layers and custom expressions that test authors have to deal with.

Also, more info in the expanded commit messages.

TODO:

• clean up commit history
• self-review the docs; make sure suggestions were applied correctly while rebasing
• AVOIDED: lib/modules: add attrArgName #177576
• lib/modules: Make unifyModuleSyntax key optional, remove extensionOffset #177157
• lib.types: Add deferredModule #163617
  • defaults option
• do something about the now-legacy code that's duplicated
  • either move to "legacy" directory and leave in place until all migrated?
  • or big bang and monitor the fallout
  • use nixos-test-staging jobset in hydra
    • before rebase
      • latest complete comparison https://hydra.nixos.org/eval/1768916?compare=1768212&full=0
    • baseline eval 2 https://hydra.nixos.org/eval/1769103
      • first comparison of 2nd rebase https://hydra.nixos.org/eval/1769104?compare=1769103&full=0
• maybe not put name in specialArgs; feasible with some network.nix refactoring to make the node config skeleton static.
• convert a couple more tests
  done:
  - _3proxy
  - acme
  - adguardhome
  - aesmd
  - agate
  - avahi
  - cassandra
  - oci-containers
  - hostname
  - networking
• docs in the manual
• do a real example of multi-dimensional matrix; have tested this manually but example was not commit-worthy
  • test with variant that has extra configuration: avahi
  • test with parameter: oci-containers
  • demonstrate 2+ orthogonal choices (networking: implementation and testcase axes)
  • (unnecessary) demonstrate how to prune irrelevant choices (matrix.<choice>.enable = false + matrix.<choice>.after)
    • with enable (cassandra.passthru.tests.nixos)
    • (unnecessary) with nesting (if that works)
• add an example that modifies the nixos config directly from matrix (as value.<value> is a "parallel universe" of the entire test config) -> avahi
• matrix: rename value to module?
• Option docs improvements #179351
• generated docs

Closes #144110

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 22.11 Release Notes (or backporting 22.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[mweinelt]

Just ❤️

[fricklerhandwerk]

This is all fun, but should rather go to your personal blog post or whatever material targeted to a select audience. Literally everyone outside the western cultural circle in their mid-30s will be completely thrown off. And possibly pissed to have to filter out what matters if they just want to get their job done.

Please, please, please, do not use cultural homages and idioms in open source software! They only entrench all the social biases we already have in the community.

(I'd love to see it kept in the Git history though, because it is funny if you understand it!)

[roberth]

I agree. I already felt it was on the edge of acceptable, but you're absolutely right about the biases. Thank you for reminding me!
In a way it was also "hiding" the fact that that part of the documentation was rather incomplete.

I haven't rebased it away, so you'll still be able to find it. (Remove movie references; hash might change)

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/specialargs-in-tests/19689/5

[roberth]

@Atemu just rebased. Annoying but wasn't all that bad. Personally I can't spend a lot of time on this, but it seems that the PR is in a decent state now.

TODO: hacking

• maybe refactor exposeIntermediateAttrs as mentioned in last commit: (nixos/lib/testing: Pull callTest in)
• extraConfig -> test <= just use the term from the domain. It's a test configuration module, so call it test.
• fix the rebase mistake that caused a test for a bad module system feature to be readded

TODO: review

• thorough review of the correctness of the rebase, especially check for subtle changes in the tests
• final review of docs

Maybe the same should be done for exposeIntermediateAttrs. It'd turn
the two functions into a catamorphism that's integrated into the
modules.

TODO: final double-check; manual testing

• nixosTests
• nixosTest
• nixos.lib.*
• testing-python.nix/makeTest
• nixos/release-combined.nix/nixos.tests

[Atemu]

I think we should do our best to get an MVP of this PR without major regressions in first. Any further niceties and minor refactorings can come later. It's important further changes to tests can make use of this PR's core changes ASAP so that we don't diverge endlessly.

Would a hydra jobset to verify help?

[roberth]

Just need to do a bit of due diligence, but yeah, no need to drag this out.

Would a hydra jobset to verify help?

There's a jobset any nixpkgs maintainer can reuse: https://hydra.nixos.org/jobset/nixos/python-test-refactoring. I'll document how to use it effectively.

[roberth]

Would a hydra jobset to verify help?

Documenting the testing process here #216660

[roberth]

I have to be honest: I don't think I'll get around to completing this. It'd be great if someone could take over.

Also: don't use the existence of this as an excuse not to write parameterized nixos tests. The true cost of writing a test is in finding the requirements and doing implementation work. Changing an already parameterized test to use a framework like this is tiny.

[roberth]

This needs modularization.

In the current code, it adds a hook to override the return value. This isn't terrible, but obscures the fact that conceptually the test matrix comes before the test, even if both things can be specified in the same module. It also makes it impossible to use the test matrix logic in a context other than the NixOS test framework, or in a combined way.

TODO:

• Simplify
  • One option / attr to retrieve release.nix attributes (Hydra)
  • One option / attr to retrieve nix-build attributes (also OfBorg and nix build with legacyPackages)
  • One option / attr to retrieve flattened attributes (e.g. flake checks)
• Extraction
  • Factor out the matrix module. I think this is almost done thanks to it being a separate file
  • Document the matrix system
  • Test the matrix evaluation independent of the rest of the test framework
• Integration
  • Use it with the test framework. imports the test framework into it?
  • Use it with simple build tests, such as the etc tests
  • Update all-tests.nix to use it
  • Apply it to the entire all-tests.nix? A test suite is just a test matrix where one of the axes is a set of test cases. (And the matrix (or tensor) does not have to be rectangular, boxy, or whatever math term we want to abuse for it. Not all coordinates have to exist. The shape can be irregular. Etc etc.)
    • can be used for e.g. running all tests with busybox

EDIT: Also move it to lib, or modules.generic.matrix in the flake? There's nothing NixOS-specific about the refactored test matrix concept and module.

[AleXoundOS]

Is it true that without merging this PR currently
one cannot override arbitrary parts of the nodes attribute set
of a test from nixos/tests?

My attempts have failed.

If I import `nixos/tests/some-test.nix`, the resulting set is not suitable for reuse in `runTest`. NixOS tests import `make-test-python.nix` right on the first line, that doesn't allow to access the original `nodes` attribute set.