"Inappropriate ioctl for device" running zpool create

[mcwhitfield]

Hi,

I'm on an existing NixOS installation on a host system, and trying to configure a disk to be connected to a new system (SSD connected to host via SATA-USB adapter). It's a ZFS configuration with ephemeral root for use with Impermanence. Disko fails at zpool creation with a pretty cryptic low-level error:

+ zpool create -f zpool-rpi-0-0 -R /mnt -O acltype=posixacl -O canmount=off -O compression=on -O encryption=on -O keyformat=passphrase -O keylocation=file:///tmp/secret.key -O mountpoint=none -O xattr=sa /dev/disk/by-partlabel/disk-primary-zfs
internal error: cannot create 'zpool-rpi-0-0': Inappropriate ioctl for device
qemu: uncaught target signal 6 (Aborted) - core dumped

However, immediately after disko fails, running the exact same zpool create command in my shell works just fine.

I tried looking into this "inappropriate ioctl for device," but that error comes from a syscall somewhere way down deep, with no way that I can find to really narrow down the source. I can't find any examples of this error occurring with any relation to ZFS or disko. Given the qemu presence in the error message, I guess maybe this is somehow related to virtualization?

In case it was ZFS related, I've tinkered around with some of the options there (in particular setting the encryption key interactively), or even just using one of the example ZFS configs directly, and I see the same error. I also tried disko.enableConfig = false based on this doc, but it doesn't explain what that option is, or why I'd want to disable it, and the option description isn't much clearer. At any rate, it doesn't affect the issue.

---

Full output:

❯ sudo disko --mode disko --flake /etc/nixos#rpi-0-0 --debug
+ shift
+ [[ 0 -gt 0 ]]
+ [[ disko = \f\o\r\m\a\t ]]
+ [[ disko = \m\o\u\n\t ]]
+ [[ disko = \d\i\s\k\o ]]
+ [[ -n x ]]
+ [[ /etc/nixos#rpi-0-0 =~ ^(.*)#([^#"]*)$ ]]
+ flake=/etc/nixos
+ flakeAttr=rpi-0-0
+ [[ -z rpi-0-0 ]]
+ nix_args+=("--arg" "flake" "\"$flake\"")
+ nix_args+=("--argstr" "flakeAttr" "$flakeAttr")
+ nix_args+=(--extra-experimental-features flakes)
++ nix-build /nix/store/xfagkkxikvn3wnai4rnmwxqpvryklb9z-disko/share/disko/cli.nix --no-out-link --impure --argstr mode disko --arg flake '"/etc/nixos"' --argstr flakeAttr rpi-0-0 --extra-experimental-features flakes
trace: true
this derivation will be built:
  /nix/store/rd4k91zvhafk291r067fg5f72kx5n5dw-disko.drv
building '/nix/store/rd4k91zvhafk291r067fg5f72kx5n5dw-disko.drv'...
+ script=/nix/store/9psl5l0g13caayip6x8hjs0idpgl0agb-disko
+ [[ -n '' ]]
+ exec /nix/store/9psl5l0g13caayip6x8hjs0idpgl0agb-disko
umount: /mnt: not found
++ realpath /dev/disk/by-id/ata-Samsung_SSD_870_QVO_1TB_S5RRNF0W311527E
+ disk=/dev/sdc
+ lsblk -a -f
NAME        FSTYPE     FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
loop0
loop1
loop2
loop3
loop4
loop5
loop6
loop7
sda
sdb
sdc
├─sdc1      vfat       FAT32       5EE0-C543
└─sdc2
nvme0n1
├─nvme0n1p1 vfat       FAT32       1562-7E09                             478.7M     6% /boot
└─nvme0n1p2 zfs_member 5000  zpool 7895610550053407653
+ lsblk --output-all --json
+ bash -x
++ dirname /nix/store/r3xam6azp7zcd8a2ybwqdnqv0iliggl1-disk-deactivate/disk-deactivate
+ jq -r --arg disk_to_clear /dev/sdc -f /nix/store/r3xam6azp7zcd8a2ybwqdnqv0iliggl1-disk-deactivate/disk-deactivate.jq
+ set -fu
+ wipefs --all -f /dev/sdc1
/dev/sdc1: 8 bytes were erased at offset 0x00000052 (vfat): 46 41 54 33 32 20 20 20
/dev/sdc1: 1 byte was erased at offset 0x00000000 (vfat): eb
/dev/sdc1: 2 bytes were erased at offset 0x000001fe (vfat): 55 aa
++ zdb -l /dev/sdc2
++ sed -nr 's/ +name: '\''(.*)'\''/\1/p'
+ zpool=zpool-rpi-0-0
+ [[ -n zpool-rpi-0-0 ]]
+ zpool destroy -f zpool-rpi-0-0
internal error: cannot destroy 'zpool-rpi-0-0': Inappropriate ioctl for device
qemu: uncaught target signal 6 (Aborted) - core dumped
bash: line 4: 2434218 Aborted                 (core dumped) zpool destroy -f "$zpool"
+ unset zpool
+ wipefs --all -f /dev/sdc2
++ zdb -l /dev/sdc
++ sed -nr 's/ +name: '\''(.*)'\''/\1/p'
+ zpool=zpool
+ [[ -n zpool ]]
+ zpool destroy -f zpool
could not destroy 'zpool': could not unmount datasets
+ unset zpool
+ wipefs --all -f /dev/sdc
/dev/sdc: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
/dev/sdc: 8 bytes were erased at offset 0xe8e0db5e00 (gpt): 45 46 49 20 50 41 52 54
/dev/sdc: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
+ dd if=/dev/zero of=/dev/sdc bs=440 count=1
1+0 records in
1+0 records out
440 bytes copied, 0.000254876 s, 1.7 MB/s
+ lsblk -a -f
NAME        FSTYPE     FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
loop0
loop1
loop2
loop3
loop4
loop5
loop6
loop7
sda
sdb
sdc
nvme0n1
├─nvme0n1p1 vfat       FAT32       1562-7E09                             478.7M     6% /boot
└─nvme0n1p2 zfs_member 5000  zpool 7895610550053407653
++ mktemp -d
+ disko_devices_dir=/tmp/tmp.YGK6fasXA5
+ trap 'rm -rf "$disko_devices_dir"' EXIT
+ mkdir -p /tmp/tmp.YGK6fasXA5
+ device=/dev/disk/by-id/ata-Samsung_SSD_870_QVO_1TB_S5RRNF0W311527E
+ imageSize=2G
+ name=primary
+ type=disk
+ device=/dev/disk/by-id/ata-Samsung_SSD_870_QVO_1TB_S5RRNF0W311527E
+ type=gpt
+ sgdisk --set-alignment=2048 --align-end --new=1:0:+512M --change-name=1:disk-primary-ESP --typecode=1:EF00 /dev/disk/by-id/ata-Samsung_SSD_870_QVO_1TB_S5RRNF0W311527E
Creating new GPT entries in memory.
The operation has completed successfully.
+ partprobe /dev/disk/by-id/ata-Samsung_SSD_870_QVO_1TB_S5RRNF0W311527E
+ udevadm trigger --subsystem-match=block
+ udevadm settle
+ device=/dev/disk/by-partlabel/disk-primary-ESP
+ extraArgs=()
+ declare -a extraArgs
+ format=vfat
+ mountOptions=('defaults')
+ declare -a mountOptions
+ mountpoint=/boot
+ type=filesystem
+ mkfs.vfat /dev/disk/by-partlabel/disk-primary-ESP
mkfs.fat 4.2 (2021-01-31)
+ sgdisk --set-alignment=2048 --align-end --new=2:0:-0 --change-name=2:disk-primary-zfs --typecode=2:8300 /dev/disk/by-id/ata-Samsung_SSD_870_QVO_1TB_S5RRNF0W311527E
The operation has completed successfully.
+ partprobe /dev/disk/by-id/ata-Samsung_SSD_870_QVO_1TB_S5RRNF0W311527E
+ udevadm trigger --subsystem-match=block
+ udevadm settle
+ device=/dev/disk/by-partlabel/disk-primary-zfs
+ pool=zpool-rpi-0-0
+ type=zfs
+ echo /dev/disk/by-partlabel/disk-primary-zfs
+ device=none
+ fsType=tmpfs
+ mountOptions=('defaults' 'size=6G' 'mode=755')
+ declare -a mountOptions
+ mountpoint=/
+ type=nodev
+ mode=
+ mountOptions=('defaults')
+ declare -a mountOptions
+ mountpoint=
+ name=zpool-rpi-0-0
+ options=()
+ declare -A options
+ rootFsOptions=(['acltype']='posixacl' ['canmount']='off' ['compression']='on' ['encryption']='on' ['keyformat']='passphrase' ['keylocation']='file:///tmp/secret.key' ['mountpoint']='none' ['xattr']='sa')
+ declare -A rootFsOptions
+ type=zpool
+ readarray -t zfs_devices
++ cat /tmp/tmp.YGK6fasXA5/zfs_zpool-rpi-0-0
+ zpool create -f zpool-rpi-0-0 -R /mnt -O acltype=posixacl -O canmount=off -O compression=on -O encryption=on -O keyformat=passphrase -O keylocation=file:///tmp/secret.key -O mountpoint=none -O xattr=sa /dev/disk/by-partlabel/disk-primary-zfs
internal error: cannot create 'zpool-rpi-0-0': Inappropriate ioctl for device
qemu: uncaught target signal 6 (Aborted) - core dumped
/nix/store/9psl5l0g13caayip6x8hjs0idpgl0agb-disko: line 149: 2434486 Aborted                 (core dumped) zpool create -f zpool-rpi-0-0 -R /mnt -O acltype=posixacl -O canmount=off -O compression=on -O encryption=on -O keyformat=passphrase -O keylocation=file:///tmp/secret.key -O mountpoint=none -O xattr=sa "${zfs_devices[@]}"
+ rm -rf /tmp/tmp.YGK6fasXA5

    disko.devices = {
      nodev."/" = {
        mountpoint = "/";
        device = "none";
        fsType = "tmpfs";
        mountOptions = ["defaults" "size=${cfg.tmpfs.maxSize}" "mode=755"];
      };
      disk.primary = {
        type = "disk";
        device = cfg.disk;
        content = {
          type = "gpt";
          partitions = {
            ESP = {
              size = "512M";
              type = "EF00";
              content = {
                type = "filesystem";
                format = "vfat";
                mountpoint = "/boot";
              };
            };
            zfs = {
              size = "100%";
              content = {
                inherit pool;
                type = "zfs";
              };
            };
          };
        };
      };
      zpool.${pool} = {
        type = "zpool";
        rootFsOptions = {
          encryption = "on";
          compression = "on";
          xattr = "sa";
          acltype = "posixacl";
          keyformat = "passphrase";
          keylocation = "file:///tmp/secret.key";
          canmount = "off";
          mountpoint = "none";
        };
        postCreateHook = ''
          zfs set keylocation="prompt" "${pool}";
        '';
        datasets = let
          volume = mountpoint: {
            type = "zfs_fs";
            options = {
              inherit mountpoint;
              "com.sun:auto-snapshot" = "true";
            };
          };
        in {
          nix = volume "/nix";
          persist = volume "/persist";
          "persist/home" = volume "/persist/home";
        };
      };
    };
  };

[iFreilicht]

The enableConfig option should definitely be documented better. Apart from that, did you see #700? Could it be that you're experiencing the same issue?