nixos/doc: Add Developing the Test Driver #216660
Conversation
|
|
||
| As we currently have some flaky tests, newly failing tests are expected, but should be reviewed to make sure that | ||
| - The number of failures did not increase significantly. | ||
| - All failures that do occur can reasonably be assumed to fail for a different reason than the changes. |
There was a problem hiding this comment.
The document itself is good.
At the same time, what do you think about rigorously dropping everything what's flaky? Flaky tests are a nuisance to everyone who invests good work to get changes through and have the side-effect of wearing down motivation to painstaking.
There was a problem hiding this comment.
As long as we don't mistake flaky Hydra for flaky tests, this would absolutely be a win.
We could perhaps move them to a flaky attrset, so that they can be fixed and moved back when they have a record of being reliable again.
Reasons to prefer dropping over moving:
- less is more
- satisfying
Reasons to prefer moving:
- the state of the test remains visible, so that someone can pick it up and debug it
- some tests need to evolve to a non-flaky state, as race conditions are discovered
- no loss of coverage
- Hermit
Hermit emulates the scheduler deterministically. This should let us find debuggable counterexamples, I think. Might be too new/experimental, and might not support KVM. We'll want to run it only on the flaky tests, because it is slower by an order of magnitude, but NixOS isn't poor so that's not a reason to avoid it.
So yeah, I think moving those tests would be great!
There was a problem hiding this comment.
Having different sets of which only one is really mandatory for contributors to satisfy sounds like a great improvement, yes.
There was a problem hiding this comment.
Opened #216828 for this.
There was a problem hiding this comment.
FWIW, @JulienMalka wanted to explore Hermit usage in NixOS tests, I don't know if he has something to share, but just mentioning him as it can be interesting.
There was a problem hiding this comment.
Indeed I think it'd be very interesting to deep a little further into Hermit, see what improvements in test reproducibility we can get and at what cost. I'll certainly work on that subject soon, available to chat more about it.
There was a problem hiding this comment.
What parts of the nixos test driver stack would be touched by using hermit? My gut feel says that this would be "just" configuring the guest vm's nixos config with kernel patch + config properly, is that about right?
There was a problem hiding this comment.
Ideally Hermit supports KVM, so that we only have to wrap the top level invocation in nixos/lib/testing/run.nix. Otherwise, we have a qemu invocation in nixos/modules/virtualisation/qemu-vm.nix, which is included (a bit indirectly) from nixos/lib/testing/nodes.nix. Configuring the kernel works the way it normally does in NixOS, I think.
roberth
force-pushed
the
nixos-doc-develop-test-driver
branch
from
fb07a3c
to
9964891
Compare
|
@ofborg eval |
Description of changes
Describe how to work on the NixOS Test Driver.
Main reason to write this was the test process.
Things done
sandbox = trueset innix.conf? (See Nix manual)nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/)