Introduction

Why use nix-unit?

• Simple structure compatible with lib.debug.runTests

• Allows individual test attributes to fail individually.

Rather than evaluating the entire test suite in one go, serialise & compare nix-unit uses the Nix evaluator C++ API. Meaning that we can catch test failures individually, even if the failure is caused by an evaluation error.

• Fast.

No additional processing and coordination overhead caused by the external process approach.

Simple

In it's simplest form a nix-unit test suite is just an attribute set where test attributes are prefix with test. Test attribute sets contain the keys expr, expressing the test & expected, expressing the expected results.

An expression called test.nix containing:

{
  testPass = {
    expr = 1;
    expected = 1;
  };

  testFail = {
    expr = { x = 1; };
    expected = { y = 1; };
  };

  testFailEval = {
    expr = throw "NO U";
    expected = 0;
  };
}

Evaluated with nix-unit:

$ nix-unit test.nix

Results in the output:

❌ testFail
{ x = 1; } != { y = 1; }

☢️ testFailEval
error:
       … while calling the 'throw' builtin

         at /home/adisbladis/nix-eval-jobs/test.nix:13:12:

           12|   testFailEval = {
           13|     expr = throw "NO U";
             |            ^
           14|     expected = 0;

       error: NO U

✅ testPass

😢 1/3 successful
error: Tests failed

Flakes

flake.nix

Building on top of the simple classic example the same type of structure could also be expressed in a flake.nix:

{
  description = "A very basic flake using nix-unit";

  outputs = { self, nixpkgs }: {
    libTests = {
      testPass = {
        expr = 1;
        expected = 1;
      };
    };
  };
}

And is evaluated with nix-unit like so:

$ nix-unit --flake '.#libTests'

flake checks

You can also use nix-unit in flake checks (link).

Create a tests and checks outputs.

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs";
    nix-unit.url = "github:nix-community/nix-unit";
    nix-unit.inputs.nixpkgs.follows = "nixpkgs";
  };
  outputs = { self, nixpkgs, nix-unit, ... }:
    let
      forAllSystems = nixpkgs.lib.genAttrs [
        "x86_64-linux"
        "aarch64-linux"
        "x86_64-darwin"
        "x86_64-windows"
      ];
    in
    {
      tests.testPass = { expr = 3; expected = 4; };

      checks = forAllSystems (system: {
        default = nixpkgs.legacyPackages.${system}.runCommand "tests"
          {
            nativeBuildInputs = [ nix-unit.packages.${system}.default ];
          } ''
          export HOME="$(realpath .)"
          # The nix derivation must be able to find all used inputs in the nix-store because it cannot download it during buildTime.
          nix-unit --eval-store "$HOME" \
            --extra-experimental-features flakes \
            --override-input nixpkgs ${nixpkgs} \
            --flake ${self}#tests
          touch $out
        '';
      });
    };
}

Run nix flake check and get an error as expected.

error: builder for '/nix/store/73d58ybnyjql9ddy6lr7fprxijbgb78n-nix-unit-tests.drv' failed with exit code 1;
       last 10 log lines:
       > /build/nix-20-1/expected.nix --- 1/2 --- Nix
       > 1 3
       >
       > /build/nix-20-1/expected.nix --- 2/2 --- Nix
       > 1 4
       >
       >
       >
       > 😢 0/1 successful
       > error: Tests failed
       For full logs, run 'nix log /nix/store/73d58ybnyjql9ddy6lr7fprxijbgb78n-nix-unit-tests.drv'.

Trees

While simple flat attribute sets works you might want to express your tests as a deep attribute set. When nix-unit encounters an attribute which name is not prefixed with test it recurses into that attribute to find more tests.

Example:

{
  testPass = {
    expr = 1;
    expected = 1;
  };

  testFail = {
    expr = { x = 1; };
    expected = { y = 1; };
  };

  testFailEval = {
    expr = throw "NO U";
    expected = 0;
  };

  nested = {
    testFoo = {
      expr = "bar";
      expected = "bar";
    };
  };
}

Results in the output:

✅ nested.testFoo
❌ testFail
/run/user/1000/nix-244499-0/expected.nix --- Nix
1 { x = 1; }                                         1 { y = 1; }


☢️ testFailEval
error:
       … while calling the 'throw' builtin

         at /home/adisbladis/sauce/github.com/nix-community/nix-unit/trees.nix:13:12:

           12|   testFailEval = {
           13|     expr = throw "NO U";
             |            ^
           14|     expected = 0;

       error: NO U

✅ testPass

😢 2/4 successful
error: Tests failed

Testing errors

While testing the happy path is a good start, you might also want to verify that expressions throw the error you expect. You check for a specifc type of error by setting expectedError.type and/or use expectedError.msg to search its message for the given regex.

Example:

tests/default.nix

{
  testCatchMessage = {
    expr = throw "10 instead of 5";
    expectedError.type = "ThrownError";
    expectedError.msg = "\\d+ instead of 5";
  };
}

->

✅ testCatchMessage

🎉 1/1 successful

Note: Regular expression like the one above are supported

Supported error types

The following values for expectedError.type are valid:

• RestrictedPathError
• MissingArgumentError
• UndefinedVarError
• TypeError
• Abort
• ThrownError
• AssertionError
• ParseError
• EvalError

Coverage

lib.Coverage.addCoverage

Generate coverage testing for public interfaces.

public

: The public interface to generate coverage for

tests

: Attribute set of tests to match agains

::: {.example #function-library-example-lib.Coverage.addCoverage}

lib.Coverage.addCoverage usage example

# Expression
let
  # The public interface (attrset) we are testing
  public = {
    addOne = x: x + 1;
  };
  # Test suite
  tests = {
    addOne = {
      testAdd = {
        expr = public.addOne 1;
        expected = 2;
      };
    };
  };
in addCoverage public tests

# Returns
{
  addOne = {
    testAdd = {
      expected = 2;
      expr = 2;
    };
  };
  coverage = {
    testAddOne = {
      expected = true;
      expr = true;
    };
  };
}

:::

Hacking

This document outlines hacking on nix-unit itself.

Getting started

To start hacking run either nix-shell (stable Nix) nix develop (Nix Flakes).

Then create the meson build directory:

$ meson build
$ cd build

And use ninja to build:

$ ninja

Formatter

Before submitting a PR format the code with nix fmt and ensure Flake checks pass with nix flake check.

FAQ

What about a watch mode?

This adds a lot of additional complexity and for now is better dealt with by using external file watcher tools such as Reflex & Watchman.

Can I change the colors?

nix-unit uses difftastic, which can be configured via environment variables. You can turn off colors via DFT_COLOR=never, give difftastic a hint for choosing better colors with DFT_BACKGROUND=light or see the full list of options via e.g. nix run nixpkgs#difftastic -- --help.

Comparison with other tools

This comparison matrix was originally taken from Unit test your Nix code but has been adapted. Pythonix is excluded as it's unmaintained.

1. Snapshot testing