System Requirements: nixos-anywhere

Install NixOS everywhere via ssh

Documentation Index

Requirements

Source Machine:

1. Supported Systems:

   • Linux or macOS computers with Nix installed.
   • NixOS
   • Windows systems using WSL2.

   Note: Should be able to build nix derivations for the target platform. Otherwise --build-on-remote can be used.

2. Nix Installation: If Nix is not yet installed on your system, refer to the nix installation page.

Destination Machine:

The machine must be reachable over the public internet or local network. Nixos-anywhere does not support wifi networks. If a VPN is needed, define a custom installer via the --kexec flag which connects to your VPN.

1. Direct Boot Option:

   • Must be already running a NixOS installer.

2. Alternative Boot Options: If not booting directly from a NixOS installer image:

   • Architecture & Support: Must be operating on:
     • x86-64 or aarch64 Linux systems with kexec support. Note: While most x86-64 Linux systems support kexec, if you're using an architecture other than those mentioned, you may need to specify a different kexec image manually.
   • Memory Requirements:
     • At least 1.5 GB of RAM (excluding swap space).

Quickstart Guide: nixos-anywhere

Install NixOS everywhere via ssh

Documentation Index

Introduction

This guide documents a simple installation of NixOS using nixos-anywhere on a target machine running x86_64 Linux with kexec support. The example used in this guide installs NixOS on a Hetzner cloud machine. The configuration may be different for some other instances. We will be including further examples in the How To Guide as and when they are available.

You will need:

• A flake that controls the actions to be performed
• A disk configuration containing details of the file system that will be created on the new server.
• A target machine, reachable via SSH, with your SSH public key deployed and and the privilege to either login directly as root or to use password-less sudo.

nixos-anywhere doesn’t need to be installed. You can run it directly from Numtide's repository on Github.

Details of the flake, the disk configuration and the CLI command are discussed below.

Steps required to run nixos-anywhere

1. Enable Flakes:

   Ensure that flakes are enabled on your system. To enable flakes, refer to the NixOS Wiki.

2. Initialize a Flake:

   The easiest way to start is to copy our example flake.nix into a new directory. This example is tailored for a virtual machine setup similar to one on Hetzner Cloud, so you might need to adapt it for your setup.

   Hardware-Specific Configuration: If you're not using a virtual machine, you'll need to generate a custom hardware configuration with nixos-generate-config.

• Get nixos-generate-config onto the Target Machine:

  1. Option 1: If NixOS is not installed, boot into an installer without first installing NixOS.
  2. Option 2: Use the kexec tarball method, as described here.

• Generate Configuration: Run the following command on the target machine:

  nixos-generate-config --no-filesystems --root /mnt
  

  This creates the necessary configuration files under /mnt/etc/nixos/, which you can then customize as needed and copy them to your local machine in order to include them in your flake.

3. Find SSH Key Line:
   if you cloned our nixos-anywhere-example you will also replace the SSH key like this: In your configuration, locate the line that reads:

   # change this to your ssh key
               "CHANGE"
   

   Replace the text CHANGE with your own SSH key. This is crucial, as you will not be able to log into the target machine post-installation without it.

4. In the same directory, create a file named disk-config.nix. This will be used to specify the disk layout to the disko tool, which nixos-anywhere uses to partition, format and mount the disks. Again, for a simple installation you can paste the contents from the example here. This configures a standard GPT (GUID Partition Table) partition compatible with both EFI and BIOS systems, and mounts the disk as /dev/sda. If this doesn’t meet your requirements, choose an example that suits your disk layout from the disko examples. For more information about this configuration, refer to the disko documentation.

5. Run the following command to create the flake.lock file:

   nix flake lock
   

   Optionally, you can commit these files to a repo such as Github, or you can simply reference your local directory when you run nixos-anywhere. This example uses a local directory on the source machine.

6. On the target machine, make sure you have access as root via ssh by adding your SSH key to the file authorized_keys in the directory /root/.ssh

   Optionally, bootstrapping can also be performed through password login. For example through the image-installer-* provided by nix-community/nixos-images. Assign your password to the SSH_PASS environment variable and specify --env-password as an additional command line option. This will provide ssh-copy-id with the required password.

7. (Optional) Test your nixos and disko configuration:

   The following command will automatically test your nixos configuration and run disko inside a virtual machine, where

   • <path to configuration> is the path to the directory or repository containing flake.nix and disk-config.nix

   • <configuration name> must match the name that immediately follows the text nixosConfigurations. in the flake, as indicated by the comment in the example).

   nix run github:nix-community/nixos-anywhere -- --flake <path to configuration>#<configuration name> --vm-test
   

8. You can now run nixos-anywhere from the command line as shown below, where:

   • <path to configuration> is the path to the directory or repository containing flake.nix and disk-config.nix

   • <configuration name> must match the name that immediately follows the text nixosConfigurations. in the flake, as indicated by the comment in the example).

   • <ip address> is the IP address of the target machine.

     nix run github:nix-community/nixos-anywhere -- --flake <path to configuration>#<configuration name> root@<ip address>
     

     The command would look  like this if you had created your files in a directory named /home/mydir/test and the IP address of your target machine is 37.27.18.135:

     nix run github:nix-community/nixos-anywhere -- --flake /home/mydir/test#hetzner-cloud root@37.27.18.135
     

     nixos-anywhere will then run, showing various output messages at each stage. It may take some time to complete, depending on Internet speeds. It should finish by showing the messages below before returning to the command prompt.

     Installation finished. No error reported.
     Warning: Permanently added '<ip-address>' (ED25519) to the list of known hosts
     

     When this happens, the target server will have been overwritten with a new installation of NixOS. Note that the server's public SSH key will have changed.

     If you have previously accessed this server using SSH, you may see the following message the next time you try to log in to the target.

     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
     It is also possible that a host key has just been changed.
     The fingerprint for the ED25519 key sent by the remote host is
     XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.
     Please contact your system administrator.
     Add correct host key in ~/.ssh/known_hosts to get rid of this message.
     Offending ECDSA key in ~/.ssh/known_hosts:6
       remove with:
       ssh-keygen -f ~/.ssh/known_hosts" -R "<ip address>"
     Host key for <ip_address> has changed and you have requested strict checking.
     Host key verification failed.
     

     This is because the known_hosts file in the .ssh directory now contains a mismatch, since the server has been overwritten. To solve this, use a text editor to remove the old entry from the known_hosts file. The next connection attempt will then treat this as a new server.

     The error message line Offending ECDSA key in ~/.ssh/known_hosts: gives the line number that needs to be removed from the known_hosts file.

     The new server's configurations are defined in the flake. nixos-anywhere does not create etc/nixos/configuration.nix, since it expects the server to be administered remotely. Any future changes to the configuration should be made to the flake, and you would reference this flake when doing the nixos-rebuild command or a deployment tool of your choice i.e. colmena, nixinate.

     This example can be run from the machine itself for updating (replace <URL to your flake> with your flake i.e. .# if your flake is in the current directory):

     nixos-rebuild switch --flake <URL to your flake>
     

     You can also run nixos-rebuild to update a machine remotely, if you have set up an openssh server and your ssh key for the root user:

     nixos-rebuild switch --flake <URL to your flake> --target-host "root@<ip address>"
     

     For more information on different use cases of nixos-anywhere please refer to the How to Guide, and for more technical information and explanation of known error messages, refer to the Reference Manual.

How To Guide: nixos-anywhere

Install NixOS everywhere via ssh

Documentation Index

Contents

Installing on a machine with no operating system

Using your own kexec image

Secrets and full disk encryption

Use without flakes

Terraform

Nix-channels / NIX_PATH

IPv6-only targets

Installing on a machine with no operating system

If your machine doesn't currently have an operating system installed, you can still run nixos-anywhere remotely to automate the install. To do this, you would first need to boot the target machine from the standard NixOS installer. You can either boot from a USB or use netboot.

The NixOS installation guide has detailed instructions on how to boot the installer.

When you run nixos-anywhere, it will determine whether a NixOS installer is present by checking whether the /etc/os-release file contains the identifier VARIANT_ID=installer. This identifier is available on releases NixOS 23.05 or later.

If an installer is detected, nixos-anywhere will not attempt to kexec into its own image. This is particularly useful for targets that don't have enough RAM for kexec or don't support kexec.

NixOS starts an SSH server on the installer by default, but you need to set a password in order to access it. To set a password for the nixos user, run the following command in a terminal on the NixOS machine:

passwd

If you don't know the IP address of the installer on your network, you can find it by running the following command:

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    altname ens3
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute eth0
       valid_lft 86385sec preferred_lft 75585sec
    inet6 fec0::5054:ff:fe12:3456/64 scope site dynamic mngtmpaddr noprefixroute
       valid_lft 86385sec preferred_lft 14385sec
    inet6 fe80::5054:ff:fe12:3456/64 scope link
       valid_lft forever preferred_lft forever

This will display the IP addresses assigned to your network interface(s), including the IP address of the installer. In the example output below, the installer's IP addresses are 10.0.2.15, fec0::5054:ff:fe12:3456, and fe80::5054:ff:fe12:3456%eth0:

To test if you can connect and your password works, you can use the following SSH command (replace the IP address with your own):

ssh -v nixos@fec0::5054:ff:fe12:3456

You can then use the IP address to run nixos-anywhere like this:

nix run github:nix-community/nixos-anywhere -- --flake '.#myconfig' nixos@fec0::5054:ff:fe12:3456

This example assumes a flake in the current directory containing a configuration named myconfig.

Using your own kexec image

By default, nixos-anywhere downloads the kexec image from the NixOS images repository.

However, you can provide your own kexec image file if you need to use a different one. This is particularly useful for architectures other than x86_64 and aarch64, since they don't have a pre-build image.

To do this, use the --kexec command line switch followed by the path to your image file. The image will be uploaded prior to execution.

Here's an example command that demonstrates how to use a custom kexec image with nixos-anywhere:

nix run github:nix-community/nixos-anywhere -- \
  --kexec "$(nix build --print-out-paths github:nix-community/nixos-images#packages.aarch64-linux.kexec-installer-nixos-unstable-noninteractive)/nixos-kexec-installer-noninteractive-aarch64-linux.tar.gz" \
  --flake 'github:your-user/your-repo#your-system' \
  root@yourip

Make sure to replace github:your-user/your-repo#your-system with the appropriate Flake URL representing your NixOS configuration.

The example above assumes that your local machine can build for aarch64 in one of the following ways:

• Natively

• Through a remote builder

• By emulating the architecture with qemu using the following NixOS configuration:

{
  boot.binfmt.emulatedSystems = [ "aarch64-linux" ];
}

Secrets and full disk encryption

The nixos-anywhere utility offers the capability to install secrets onto a target machine. This feature is particularly beneficial when you want to bootstrap secrets management tools such as sops-nix or agenix, which rely on machine-specific secrets to decrypt other uploaded secrets.

Example: Decrypting an OpenSSH Host Key with pass

In this example, we demonstrate how to use a script to decrypt an OpenSSH host key from the pass password manager and subsequently pass it to nixos-anywhere during the installation process:

#!/usr/bin/env bash

# Create a temporary directory
temp=$(mktemp -d)

# Function to cleanup temporary directory on exit
cleanup() {
  rm -rf "$temp"
}
trap cleanup EXIT

# Create the directory where sshd expects to find the host keys
install -d -m755 "$temp/etc/ssh"

# Decrypt your private key from the password store and copy it to the temporary directory
pass ssh_host_ed25519_key > "$temp/etc/ssh/ssh_host_ed25519_key"

# Set the correct permissions so sshd will accept the key
chmod 600 "$temp/etc/ssh/ssh_host_ed25519_key"

# Install NixOS to the host system with our secrets
nixos-anywhere --extra-files "$temp" --flake '.#your-host' root@yourip

Example: Uploading Disk Encryption Secrets

In a similar vein, nixos-anywhere can upload disk encryption secrets, which are necessary during formatting with disko. Here's an example that demonstrates how to provide your disk encryption password as a file or via the pass utility to nixos-anywhere:

# Write your disk encryption password to a file
echo "my-super-safe-password" > /tmp/disk-1.key

# Call nixos-anywhere with disk encryption keys
nixos-anywhere \
  --disk-encryption-keys /tmp/disk-1.key /tmp/disk-1.key \
  --disk-encryption-keys /tmp/disk-2.key <(pass my-disk-encryption-password) \
  --flake '.#your-host' \
  root@yourip

In the above example, replace "my-super-safe-password" with your actual encryption password, and my-disk-encryption-password with the relevant entry in your pass password store. Also, ensure to replace '.#your-host' and root@yourip with your actual flake and IP address, respectively.

Example: Using existing SSH host keys

If the system contains existing trusted /etc/ssh/ssh_host_* SSH host keys and certificates, nixos-anywhere can copy them in case they are necessary during installation and system activation.

nixos-anywhere --copy-host-keys --flake '.#your-host' root@yourip

This would copy /etc/ssh/ssh_host_* to /mnt after kexec but before installation, ignoring files that already exist in destination.

Use without flakes

While nixos-anywhere is designed to work optimally with Nix Flakes, it also supports the traditional approach without flakes. This document outlines how to use nixos-anywhere without relying on flakes. You will need to import the disko nixos module in your NixOS configuration and define disko devices as described in the examples.

Generate Required Store Paths

Before you can use nixos-anywhere without flakes, you'll need to manually generate the paths for the NixOS system toplevel and disk image. The paths are generated using nix-build and are necessary for executing nixos-anywhere.

Generating NixOS System Toplevel:

Execute the following command to generate the store path for the NixOS system toplevel:

nix-build -I nixos-config=/etc/nixos/configuration.nix -E '(import <nixpkgs/nixos> {}).config.system.build.toplevel'

This will output a path in /nix/store that corresponds to the system toplevel, which includes all the software and configurations for the system. Make note of this path for later use.

Generating Disk Image without Dependencies:

To generate the disk image without dependencies, execute:

nix-build -I nixos-config=/etc/nixos/configuration.nix -E '(import <nixpkgs/nixos> {}).config.system.build.diskoNoDeps'

This will also output a script path in /nix/store that will format your disk. Keep this path handy as well.

Running NixOS-Anywhere

With both paths in hand, you can execute nixos-anywhere as follows:

nixos-anywhere --store-paths /nix/store/[your-toplevel-path] /nix/store/[your-disk-image-path]

Replace [your-toplevel-path] and [your-disk-image-path] with the corresponding store paths you generated earlier.

Terraform

The nixos-anywhere terraform modules allow you to use Terraform for installing and updating NixOS. It simplifies the deployment process by integrating nixos-anywhere functionality.

Our terraform module requires the null and external provider.

You can get these by from nixpkgs like this:

nix-shell -p '(pkgs.terraform.withPlugins (p: [ p.null p.external ]))'

You can add this expression the packages list in your devshell in flake.nix or in shell.nix.

Checkout out the module reference for examples and module parameter on how to use the modules.

Nix-channels / NIX_PATH

nixos-anywhere does not install channels onto the new system by default to save time and disk space. This for example results in errors like:

(stack trace truncated; use '--show-trace' to show the full trace)

error: file 'nixpkgs' was not found in the Nix search path (add it using $NIX_PATH or -I)

at «none»:0: (source not available)

when using tools like nix-shell/nix-env that rely on NIX_PATH beeing set.

Solution 1: Set the NIX_PATH via nixos configuration (recommended)

Instead of stateful channels, one can also populate the NIX_PATH using nixos configuration instead:

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
  # ... other inputs

  outputs = { nixpkgs, ... }:
    {
      nixosConfigurations.yoursystem = nixpkgs.lib.nixosSystem {
        system = "x86_64-linux"; # adapt to your actual system
        modules = [
          # This line will populate NIX_PATH
          { nix.nixPath = [ "nixpkgs=${inputs.nixpkgs}" ]; }
          # ... other modules and your configuration.nix
        ];
      };
    };
}

Advantage: This solution will be automatically kept up-to-date everytime the flake is updated.

In your shell you will see something in your $NIX_PATH:

$ echo $NIX_PATH
/root/.nix-defexpr/channels:nixpkgs=/nix/store/8b61j28rpy11dg8hanbs2x710d8w3v0d-source

Solution 2: Manually add the channel

On the installed machine, run:

$ nix-channel --add https://nixos.org/channels/nixos-unstable nixos
$ nix-channel --update

NixOS-anywhere on IPv6-only targets

As GitHub engineers still haven't enabled the IPv6 switch, the kexec image hosted on GitHub, cannot be used unfortunately on IPv6-only hosts. However it is possible to use an IPv6 proxy for GitHub content like that:

nixos-anywhere \
  --kexec https://gh-v6.com/nix-community/nixos-images/releases/download/nixos-unstable/nixos-kexec-installer-noninteractive-x86_64-linux.tar.gz \
...

This proxy is hosted by numtide. It also works for IPv4.

Alternatively it is also possible to reference a local file:

nixos-anywhere \
  --kexec ./nixos-kexec-installer-noninteractive-x86_64-linux.tar.gz \
...

This tarball will be then uploaded via sftp to the target.

Reference Manual: nixos-anywhere

Install NixOS everywhere via ssh

Documentation Index

TODO: Populate this guide properly

Contents

Command Line Usage

Explanation of known error messages

Command Line Usage

Usage: nixos-anywhere [options] <ssh-host>

Options:

* -f, --flake <flake_uri>
  set the flake to install the system from.
* -i <identity_file>
  selects which SSH private key file to use.
* -p, --ssh-port <ssh_port>
  set the ssh port to connect with
* --ssh-option <ssh_option>
  set an ssh option
* -L, --print-build-logs
  print full build logs
* -s, --store-paths <disko-script> <nixos-system>
  set the store paths to the disko-script and nixos-system directly
  if this is give, flake is not needed
* --no-reboot
  do not reboot after installation, allowing further customization of the target installation.
* --kexec <path>
  use another kexec tarball to bootstrap NixOS
* --post-kexec-ssh-port <ssh_port>
  after kexec is executed, use a custom ssh port to connect. Defaults to 22
* --copy-host-keys
  copy over existing /etc/ssh/ssh_host_* host keys to the installation
* --stop-after-disko
  exit after disko formatting, you can then proceed to install manually or some other way
* --extra-files <file...>
  files to copy into the new nixos installation
* --disk-encryption-keys <remote_path> <local_path>
  copy the contents of the file or pipe in local_path to remote_path in the installer environment,
  after kexec but before installation. Can be repeated.
* --no-substitute-on-destination
  disable passing --substitute-on-destination to nix-copy
* --debug
  enable debug output
* --option <key> <value>
  nix option to pass to every nix related command
* --from <store-uri>
  URL of the source Nix store to copy the nixos and disko closure from
* --build-on-remote
  build the closure on the remote machine instead of locally and copy-closuring it
* --vm-test
  build the system and test the disk configuration inside a VM without installing it to the target.

Explanation of known error messages

TODO: Add additional error messages and meanings. Fill in missing explanations

This section lists known error messages and their explanations. Some explanations may refer to the following CLI syntax:

nix run github:nix-community/nixos-anywhere -- --flake <path to configuration>#<configuration name> root@<ip address>

This list is not comprehensive. It's possible you may encounter errors that originate from the underlying operating system. These should be documented in the relevant operating system manual.