[nixosTest] Luks causes: KDSKBMODE: Inappropriate ioctl for device

[Skarlett]

Whenever I run nixosConfigurations.<<machine>>.installScript.driverInteractive

The test successfully finishes the first boot, during the second boot phase, I receive & hang on.

booted_machine # kbd_mode: KDSKBMODE: Inappropriate ioctl for device

Optimistically, when Ctrl-C is sent. I receive

booted_machine # �%GPassphrase for /dev/disk/by-partlabel/disk-sda-luksroot:

{ # disko.nix
  disko.devices = {
    disk = {
      sda = {
        type = "disk";
        device = "/dev/sda";
        content = {
          type = "gpt";
          partitions = {
            gptbios = {
              priority = 1;
              type = "EF02";
              size = "1M";
            };

            boot = {
              size = "512M";
              type = "EF00";
              content = {
                type = "filesystem";
                format = "vfat";
                mountpoint = "/boot/efi";
                mountOptions = [
                  "defaults"
                ];
              };
            };
            
            luksroot = {
              size = "100%";
              content = {
                type = "luks";
                name = "luksroot";

                passwordFile = "/tmp/luks-secret.key";
                initrdUnlock = true;
                # askPassword = true;
                content = {
                  type = "zfs";
                  pool = "rpool";
                };

                preCreateHook = ''
                 dd if=/dev/urandom bs=32 count=1 2>/dev/null | sed -e 's/[[:space:]]*//' > /tmp/luks-secret.key
                '';


              };
            };
          };
        };
      };
    };

    zpool = {
      rpool = {
        type = "zpool";
        postCreateHook = "zfs list -t snapshot -H -o name | grep -E '^rpool@blank$' || zfs snapshot rpool@blank";
        rootFsOptions = {
          compression = "zstd";
          "com.sun:auto-snapshot" = "true";
          mountpoint = "none";
          canmount = "off";
        };

        datasets = {
          "core" = {
            type = "zfs_fs";
            options.mountpoint = "none";
            options.canmount = "off";
          };

          "core/personality" = {
            type = "zfs_fs";
            options.quota = "1G";
            options.mountpoint = "legacy";
            mountpoint = "/.personality";
          };

          "core/log" = {
            type = "zfs_fs";
            options.mountpoint = "legacy";
            mountpoint = "/var/log";
          };

          "ephemeral" = {
            type = "zfs_fs";
            options = {
              encryption = "aes-256-gcm";
              keyformat = "passphrase";
              keylocation = "file:///tmp/secret.key";
              "com.sun:auto-snapshot" = "false";
            };
            mountpoint = "/";

            # use this to read the key during boot
            preCreateHook = ''
              dd if=/dev/urandom bs=512 count=1 2>/dev/null | sed -e 's/[[:space:]]*//' > /tmp/secret.key
            '';

            postCreateHook = ''
              zfs set keylocation=prompt rpool/ephemeral
            '';
          };
        };
      };
    };
  };
}

[Skarlett]

related: #501 possible dup.

noted: both issues uses impermanence.

[Skarlett]

Update: Issue persists without impermanence. Issue persists without ZFS.

The blame may lay on the file-descriptor used for passing credentials in LUKS

[Skarlett]

I can confirm that LUKS password prompt in the second boot is causing Inappropriate ioctl for device

[Skarlett]

Added back encrypted zfs and zfs appears to not cause Inappropriate ioctl for device

[Skarlett]

Test also appears to break during the first boot if askPassword is set.

[Skarlett]

seems to be our main suspect.

passwordFile = "/tmp/luks-secret.key";
initrdUnlock = true;

during the second phase, we may need to use a keyfile as a work around.

[Skarlett]

  boot.initrd.secrets = {
    "/crypto_keyfile.bin" = ./snakeoil.txt;
  };

  boot.initrd.luks.devices.luksroot = {
    # device = "/dev/disk/by-partlabel/disk-sda-luksroot";
    keyFile = "/crypto_keyfile.bin";
  };

vm-test-run-disko-cafebabe-disko> machine # cp: cannot stat '/nix/store/1d7krr8m9pq360hbzcng4xq3rljham8g-source/domains/local/masterbook/snakeoil.txt': No such file or directory
vm-test-run-disko-cafebabe-disko> machine # failed to create initrd secrets!

initrd secrets are failing to copy. Reason inconclusive,

subject of investigation
https://github.com/nix-community/disko/blob/master/lib/make-disk-image.nix#L74-L76

 > nixos-anywhere --flake /etc/nixos#masterbook --vm-test --extra-files snakeoil.txt 
--vm-test is not supported with --extra-files

[Skarlett]

Solution

  boot.initrd.preDeviceCommands = ''
    echo -n "TEST!" > /crypto_keyfile.bin
  '';

  # boot.initrd.secrets = {
  #   "/crypto_keyfile.bin" = ./snakeoil.txt;
  # };

  boot.initrd.luks.devices.luksroot = {
    # device = "/dev/disk/by-partlabel/disk-sda-luksroot";
    keyFile = "/crypto_keyfile.bin";
  };

[Skarlett]

We should move forward by passing the required options as overwrites into nixosTest.

boot.initrd.secrets - should assert it cannot be used.
/dev/fd/<N> (file-descriptor containing passwordFile) should be passed to the second boot, so that boot.initrd.luks.devices.<<name>>.keyFile is consistent to it.

add advisory on the process of using encryption in test environments,

It is also important to note the use of echo -n due to cryptsetup interpretation of file

[Mic92]

We currently don't have support for secrets in nixos vm tests. Is it this what you are running?

[Skarlett]

We currently don't have support for secrets in nixos vm tests. Is it this what you are running?

I am currently debugging the use of secrets in test environments.

I suppose that's why its been a bit painful to get working. Thank you for your reply.

This issue was created with the inclusion of
https://github.com/nix-community/nixos-anywhere

To clearify for later readers. Reference to /dev/fd/<<N>> for passwordFile is in reference of that.