Introduction

Why use nix-unit?

Simple structure compatible with lib.debug.runTests
Allows individual test attributes to fail individually.

Rather than evaluating the entire test suite in one go, serialise & compare nix-unit uses the Nix evaluator C++ API. Meaning that we can catch test failures individually, even if the failure is caused by an evaluation error.

Fast.

No additional processing and coordination overhead caused by the external process approach.

Simple

In it's simplest form a nix-unit test suite is just an attribute set where test attributes are prefix with test. Test attribute sets contain the keys expr, expressing the test & expected, expressing the expected results.

An expression called test.nix containing:

{
  testPass = {
    expr = 1;
    expected = 1;
  };

  testFail = {
    expr = { x = 1; };
    expected = { y = 1; };
  };

  testFailEval = {
    expr = throw "NO U";
    expected = 0;
  };
}

Evaluated with nix-unit:

$ nix-unit test.nix

Results in the output:

❌ testFail
{ x = 1; } != { y = 1; }

☢️ testFailEval
error:
       … while calling the 'throw' builtin

         at /home/adisbladis/nix-eval-jobs/test.nix:13:12:

           12|   testFailEval = {
           13|     expr = throw "NO U";
             |            ^
           14|     expected = 0;

       error: NO U

✅ testPass

😢 1/3 successful
error: Tests failed

Flakes

flake.nix

Building on top of the simple classic example the same type of structure could also be expressed in a flake.nix:

{
  description = "A very basic flake using nix-unit";

  outputs = { self, nixpkgs }: {
    libTests = {
      testPass = {
        expr = 1;
        expected = 1;
      };
    };
  };
}

And is evaluated with nix-unit like so:

$ nix-unit --flake '.#libTests'

flake checks

Note: flake-parts can manage this for you.

You can also use nix-unit in flake checks (link).

Create a tests and checks outputs.

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs";
    nix-unit.url = "github:nix-community/nix-unit";
    nix-unit.inputs.nixpkgs.follows = "nixpkgs";
  };
  outputs =
    {
      self,
      nixpkgs,
      nix-unit,
      ...
    }:
    let
      forAllSystems = nixpkgs.lib.genAttrs [
        "x86_64-linux"
        "aarch64-linux"
        "x86_64-darwin"
        "x86_64-windows"
      ];
    in
    {
      tests.testPass = {
        expr = 3;
        expected = 4;
      };

      checks = forAllSystems (system: {
        default =
          nixpkgs.legacyPackages.${system}.runCommand "tests"
            {
              nativeBuildInputs = [ nix-unit.packages.${system}.default ];
            }
            ''
              export HOME="$(realpath .)"
              # The nix derivation must be able to find all used inputs in the nix-store because it cannot download it during buildTime.
              nix-unit --eval-store "$HOME" \
                --extra-experimental-features flakes \
                --override-input nixpkgs ${nixpkgs} \
                --flake ${self}#tests
              touch $out
            '';
      });
    };
}

Run nix flake check and get an error as expected.

error: builder for '/nix/store/73d58ybnyjql9ddy6lr7fprxijbgb78n-nix-unit-tests.drv' failed with exit code 1;
       last 10 log lines:
       > /build/nix-20-1/expected.nix --- 1/2 --- Nix
       > 1 3
       >
       > /build/nix-20-1/expected.nix --- 2/2 --- Nix
       > 1 4
       >
       >
       >
       > 😢 0/1 successful
       > error: Tests failed
       For full logs, run 'nix log /nix/store/73d58ybnyjql9ddy6lr7fprxijbgb78n-nix-unit-tests.drv'.

Flake-parts

nix-unit provides a flake-parts module for easy integration with flakes.

You can write tests in the option flake.tests and/or perSystem.nix-unit.tests.

The module then takes care of setting up a checks derivation for you.

For this to work, you may have to specify perSystem.nix-unit.inputs to make them available in the derivation. This tends to require that you flatten some of your inputs tree using follows.

Example

This example can be used with nix flake init -t github:nix-community/nix-unit#flake-parts.

{
  description = "Description for the project";

  inputs = {
    flake-parts.url = "github:hercules-ci/flake-parts";
    flake-parts.inputs.nixpkgs-lib.follows = "nixpkgs";
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    nix-unit.url = "github:nix-community/nix-unit";
    nix-unit.inputs.nixpkgs.follows = "nixpkgs";
    nix-unit.inputs.flake-parts.follows = "flake-parts";
  };

  outputs =
    inputs@{ flake-parts, ... }:
    flake-parts.lib.mkFlake { inherit inputs; } {
      imports = [
        inputs.nix-unit.modules.flake.default
      ];
      systems = [
        "x86_64-linux"
        "aarch64-linux"
        "aarch64-darwin"
        "x86_64-darwin"
      ];
      perSystem =
        { ... }:
        {
          nix-unit.inputs = {
            # NOTE: a `nixpkgs-lib` follows rule is currently required
            inherit (inputs) nixpkgs flake-parts nix-unit;
          };
          # Tests specified here may refer to system-specific attributes that are
          # available in the `perSystem` context
          nix-unit.tests = {
            "test integer equality is reflexive" = {
              expr = "123";
              expected = "123";
            };
            "frobnicator" = {
              "testFoo" = {
                expr = "foo";
                expected = "foo";
              };
            };
          };
        };
      flake = {
        # System-agnostic tests can be defined here, and will be picked up by
        # `nix flake check`
        tests.testBar = {
          expr = "bar";
          expected = "bar";
        };
      };
    };
}

Trees

While simple flat attribute sets works you might want to express your tests as a deep attribute set. When nix-unit encounters an attribute which name is not prefixed with test it recurses into that attribute to find more tests.

Example:

{
  testPass = {
    expr = 1;
    expected = 1;
  };

  testFail = {
    expr = { x = 1; };
    expected = { y = 1; };
  };

  testFailEval = {
    expr = throw "NO U";
    expected = 0;
  };

  nested = {
    testFoo = {
      expr = "bar";
      expected = "bar";
    };
  };
}

Results in the output:

✅ nested.testFoo
❌ testFail
/run/user/1000/nix-244499-0/expected.nix --- Nix
1 { x = 1; }                                         1 { y = 1; }


☢️ testFailEval
error:
       … while calling the 'throw' builtin

         at /home/adisbladis/sauce/github.com/nix-community/nix-unit/trees.nix:13:12:

           12|   testFailEval = {
           13|     expr = throw "NO U";
             |            ^
           14|     expected = 0;

       error: NO U

✅ testPass

😢 2/4 successful
error: Tests failed

Testing errors

While testing the happy path is a good start, you might also want to verify that expressions throw the error you expect. You check for a specifc type of error by setting expectedError.type and/or use expectedError.msg to search its message for the given regex.

Example:

tests/default.nix

{
  testCatchMessage = {
    expr = throw "10 instead of 5";
    expectedError.type = "ThrownError";
    expectedError.msg = "\\d+ instead of 5";
  };
}

✅ testCatchMessage

🎉 1/1 successful

Note: Regular expression like the one above are supported

Supported error types

The following values for expectedError.type are valid:

RestrictedPathError
MissingArgumentError
UndefinedVarError
TypeError
Abort
ThrownError
AssertionError
ParseError
EvalError

coverage

`lib.coverage.addCoverage`

Generate coverage testing for public interfaces.

public

: The public interface to generate coverage for

tests

: Attribute set of tests to match agains

::: {.example #function-library-example-lib.coverage.addCoverage}

`lib.coverage.addCoverage` usage example

# Expression
let
  # The public interface (attrset) we are testing
  public = {
    addOne = x: x + 1;
  };
  # Test suite
  tests = {
    addOne = {
      testAdd = {
        expr = public.addOne 1;
        expected = 2;
      };
    };
  };
in addCoverage public tests

# Returns
{
  addOne = {
    testAdd = {
      expected = 2;
      expr = 2;
    };
  };
  coverage = {
    testAddOne = {
      expected = true;
      expr = true;
    };
  };
}

:::

Hacking

This document outlines hacking on nix-unit itself.

Getting started

To start hacking run either nix-shell (stable Nix) nix develop (Nix Flakes).

Then create the meson build directory:

$ meson build
$ cd build

And use ninja to build:

$ ninja

Formatter

Before submitting a PR format the code with nix fmt and ensure Flake checks pass with nix flake check.

FAQ

What about a watch mode?

This adds a lot of additional complexity and for now is better dealt with by using external file watcher tools such as Reflex & Watchman.

Can I change the colors?

nix-unit uses difftastic, which can be configured via environment variables. You can turn off colors via DFT_COLOR=never, give difftastic a hint for choosing better colors with DFT_BACKGROUND=light or see the full list of options via e.g. nix run nixpkgs#difftastic -- --help.

Comparison with other tools

This comparison matrix was originally taken from Unit test your Nix code but has been adapted. Pythonix is excluded as it's unmaintained.

Tool	Can test eval failures	Tests defined in Nix	in nixpkgs	snapshot testing(1)
Nix-unit	yes	yes	yes	no
runTests	no	yes	yes	no
Nixt	no	yes	no	no
Namaka	no	yes	yes	yes

Snapshot testing

Nix-unit