stable ssh keys

[Sohalt]

Since /etc/ssh/ is on the tmpfs, the VM generates a new ssh key-pair every time is gets rebooted.
I tried to create a virtiofsd share for /etc/ssh to keep the keys on the host system, but that makes sshd fail (presumably because there is an issue with the symlinks for the config).
Any idea why this would fail? Symlinks should work across filesystems and the directory should be there from the moment the machine starts, no?

[Sohalt]

I was able to get stable ssh keys using something like

system.activationScripts.ensure-ssh-key-dir.text = "mkdir -p /opt/ssh";
microvm.shares = [{
        tag = "opt";
        source = "/var/lib/microvms/foo/shares/opt";
        mountPoint = "/opt";
        proto = "virtiofs";
        socket = "/tmp/virtiofs.sock";
}];
services.openssh.hostKeys = [
  { bits = 4096; path = "/opt/ssh/ssh_host_rsa_key"; type = "rsa"; }
    { path = "/opt/ssh/ssh_host_ed25519_key"; type = "ed25519"; }
];

[astro]

I have my MicroVM's whole /etc on virtiofs on ZFS for that precise reason: stable ssh host keys. I guess the conflict is with NixOS' /etc/static?

If you think your solution is better practise we should add it to the handbook.

[yangm97]

I guess truly stable ssh keys would need to be generated beforehand and applied using some secret management tool i.e. https://github.com/Mic92/sops-nix

[Sohalt]

sops-nix uses the ssh host keys to decrypt all other secrets, so it's a chicken/egg problem

[astro]

Yeah, I run my microvms with a persistent /etc, using the once-generated ssh host key for sops-nix.

I am still open to alternative ideas, especially for provisioning on a Skyflake cluster.

[yangm97]

sops-nix uses the ssh host keys to decrypt all other secrets, so it's a chicken/egg problem

I was thinking about bootstrapping using sops-nix on the host to store the vm ssh key and then mount the file secret read-only.

[c0deaddict]

I use https://github.com/nix-community/impermanence/ to "solve" this problem. It can be used not only for SSH keys but also for other persistent state. I give each MicroVM a /persist shared via virtiofs, and then bind mount the relevant dirs/files:

services.openssh = {
 hostKeys = [
   {
     path = "/persist/etc/ssh/ssh_host_ed25519_key";
     type = "ed25519";
   }
   {
     path = "/persist/etc/ssh/ssh_host_rsa_key";
     type = "rsa";
     bits = 4096;
   }
 ];
};

fileSystems."/persist".neededForBoot = mkForce true;

environment.persistence."/persist" = {
 directories = [
   "/var/lib/systemd/coredump"
   "/var/lib/nixos" # contains user/group id map
   "/var/log"
 ];

 files = [
   "/etc/machine-id"
   "/root/.bash_history"
 ];
};

A downside, is that you need to boot the VM's once for the host keys to generated. Then (re-)encrypt all the sops/agenix secrets with the hostkey and redeploy/reboot the VM for the secrets to work.

[matthew-salerno]

I've managed to do this with impermanence and agenix-rekey. Disregarding boiler plate, it looks like this:

Host

config.age = {
  secrets.myMicroVMSsh = {
    owner = "root";
    mode = "400";
    group = "root";
    path = "/etc/vm-persist/myMicrovm/etc/ssh/ssh_host_ed25519_key";
    symlink = false;
    rekeyFile = ../secrets/myMicrovmSsh.age";
    generator.script = {pkgs, file, ...}: ''
      ${pkgs.openssh}/bin/ssh-keygen -qt ed25519 -N "" -C "root@${name}" -f ${lib.escapeShellArg (lib.removeSuffix ".age" file)}
      priv=$(${pkgs.coreutils}/bin/cat ${lib.escapeShellArg (lib.removeSuffix ".age" file)})
      ${pkgs.coreutils}/bin/shred -u ${lib.escapeShellArg (lib.removeSuffix ".age" file)}
      echo "$priv"
    '';
  };
};

Also Host (but defines the VM)

{ pkgs, impermanence, lib, agenix, agenix-rekey, ... }:
{
  microvm.vms.myMicrovm = {
    inherit pkgs;
    config = {
      imports = [
        agenix.nixosModules.default
        agenix-rekey.nixosModules.default
        impermanence.nixosModules.impermanence
      ];
      microvm.shares = [
          {
            source = "/etc/vm-persist/test-microvm";
            mountPoint = "/persist";
            tag = "persist";
            proto = "virtiofs";
          }
        ];
      age.rekey = {
        hostPubkey = ../secrets/myMicrovmSsh.pub;
        masterIdentities = [ ../../yubikey-ident.pub ];
      };
      services.openssh = {
        enable = true;
        hostKeys = [
          {
            path = "/etc/ssh/ssh_host_ed25519_key";
            type = "ed25519";
          }
        ];
      };
        
      fileSystems."/persist".neededForBoot = lib.mkForce true;
      environment.persistence."/persist" = {
        files = [
          "/etc/ssh/ssh_host_ed25519_key"
        ];
      };
    };
  };
}

I edited out the parts specific to my usecase (like configuring secrets for each vm in a map) and I haven't tested this edited code, but the general idea is sound and I've got it working with on my homelab. I should add that you don't need impermanence to make this work, you could just directly use the /persist folder. Impermanence just makes things a little nicer imo.

[RooSoft]

Steps I did:

• create a microVM
• ssh into it
• created a tarball containing all of /etc/ssh
• use wormhole to warp that tarball into the host
• on the host, extracted it in a .ssh folder near where I store .img files for the same microVM's volumes
• created a readonly share from that folder, straight into /etc/ssh on the microVM

From that point on, it's stable SSH.