Haumea

Filesystem-based module system for Nix

Haumea is not related to or a replacement for NixOS modules. It is closer to the module systems of traditional programming languages, with support for file hierarchy and visibility.

In short, haumea maps a directory of Nix files into an attribute set:

From To 
├─ foo/
│  ├─ bar.nix
│  ├─ baz.nix
│  └─ __internal.nix
├─ bar.nix
└─ _utils/
   └─ foo.nix

{
  foo = {
    bar = <...>;
    baz = <...>;
  };
  bar = <...>;
}

Haumea's source code is hosted on GitHub under the MPL-2.0 license. Haumea bootstraps itself. You can see the entire implementation in the src directory.

Why Haumea?

  • No more manual imports

    Manually importing files can be tedious, especially when there are many of them. Haumea takes care of all of that by automatically importing the files into an attribute set.

  • Modules

    Haumea takes inspiration from traditional programming languages. Visibility makes it easy to create utility modules, and haumea makes self-referencing and creating fixed points a breeze with the introduction of self, super, and root.

  • Organized directory layout

    What you see is what you get. By default1, the file tree will look exactly like the resulting attribute set.

  • Extensibility

    Changing how the files are loaded is as easy as specifying a loader, and the transformer option makes it possible to extensively manipulate the tree.

➔ Getting Started

1

Unless you are doing transformer magic

Getting Started

Haumea comes with a template for a simple Nix library. You can try out the template with:

nix flake init -t github:nix-community/haumea

This will generate flake.nix and some other relevant files in the current directory. Or if you want to create a new directory for this, run:

nix flake new <dir> -t github:nix-community/haumea

You can use haumea without the template by adding it to your flake inputs:

  inputs = {
    haumea = {
      url = "github:nix-community/haumea/v0.2.2";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    nixpkgs.url = "github:nix-community/nixpkgs.lib";
  };

Haumea is pinned to a tag here so potential breaking changes in the main branch wouldn't break downstream consumers. See the Versioning chapter for information.

The rest of this chapter will be using this template.

In flake.nix, the main thing you want to look at is lib:

    lib = haumea.lib.load {
      src = ./src;
      inputs = {
        inherit (nixpkgs) lib;
      };
    };

haumea.lib.load is the main entry point of haumea. It loads a directory (./src) of Nix files1 into an attribute set. You can see the result of this by running nix eval .#lib:

{ answer = 42; }

If you open src/answer.nix, you can see that it is a lambda that returns 42.

{ lib }:

lib.id 42

The lib here is provided by the inputs option of load:

      inputs = {
        inherit (nixpkgs) lib;
      };

The lib.id 42 in answer.nix becomes nixpkgs.lib.id 42, which evaluates to 42.

By default, the file doesn't have to specify the inputs that it does not use, or even specify any inputs at all if it is not using any inputs. Both { }: 42 and 42 are valid in this case and will do exactly the same thing.

self, super, and root are special inputs that are always available. You might already be familiar with them based on the names. These names are reserved, haumea will throw an error if you try to override them.

The documentation for load explains this more thoroughly and talks about some workarounds.

checks works basically the same, just with loadEvalTests instead of load. You can run the checks with nix flake check.

1

Non-Nix files can also be loaded using matchers

Versioning

Haumea follows semantic versioning. Breaking changes can happen on the main branch at any time, so it is recommended to pin haumea to a specific tag.

A list of available versions can be found on the GitHub releases page.

API Reference

The following sections documents everything in the library.

If you are using haumea with flakes, that would be haumea.lib.

load

Source: src/load.nix

Type: { src, loader?, inputs?, transformer? } -> { ... }

Arguments:

  • src : Path

    The directory to load files from.

  • (optional) loader : ({ self, super, root, ... } -> Path -> a) | [ Matcher ]

    Loader for the files, defaults to loaders.default. It can be either a function that loads of Nix file, or a list of matchers that allows you to load any type of file.

  • (optional) inputs : { ... }

    Extra inputs to be passed to the loader.

    self, super, and root are reserved names that cannot be passed as an input. To work around that, remove them using removeAttrs, or pass them by overriding the loader.

  • (optional) transformer : (cursor : [ String ]) -> { ... } -> a or a list of transformers

    Module transformer, defaults to [ ] (no transformation). This will transform each directory module in src, including the root. cursor represents the position of the directory being transformed, where [ ] means root and [ "foo" "bar" ] means root.foo.bar.

Files found in src are loaded into an attribute set with the specified loader. As an example, the entirety of haumea's API is loaded from the src directory.

For a directory like this:

src
├─ foo/
│  ├─ bar.nix
│  ├─ baz.nix
│  └─ __internal.nix
├─ bar.nix
└─ _utils/
   └─ foo.nix

The output will look like this:

{
  foo = {
    bar = <...>;
    baz = <...>;
  };
  bar = <...>;
}

Notice that there is no utils. This is because files and directories that start with _ are only visible inside the directory being loaded, and will not be present in the final output.

Similarly, files and directories that start with __ are only visible if they are in the same directory, meaning foo/__internal.nix is only accessible if it is being accessed from within foo.

By default, the specified inputs, in addition to self, super, and root, will be passed to the file being loaded, if the file is a function.

  • self represents the current file.
  • super represents the directory the file is in.
  • root represents the root of the src directory being loaded.

Continuing the example above, this is the content of foo/bar.nix (super and root are unused, they are just here for demonstration purposes):

{ self, super, root }:

{
  a = 42;
  b = self.a * 2;
}

self.a will be 42, which will make b 84. Accessing self.b here would cause infinite recursion, and accessing anything else would fail due to missing attributes.

super will be { bar = self; baz = <...>; internal = <...>; }.

And root will be:

{
  # foo = super;
  foo = {
    bar = <...>;
    baz = <...>;
    internal = <...>;
  };
  baz = <...>;
  utils.foo = <...>;
}

Note that this is different from the return value of load. foo.internal is accessible here because it is being accessed from within foo. Same for utils, which is accessible from all files within src, the directory being loaded.

loadEvalTests

Source: src/loadEvalTests.nix

Type: { src, loader?, inputs? } -> { }

A wrapper around load to run eval tests using runTests.

The accepted arguments are exactly the same as load.

This function will throw an error if at least one test failed, otherwise it will always return { } (an empty attribute set).

As an example, haumea's tests are loaded with loadEvalTests.

Alternatively, namaka provides utilities for snapshot testing, which can save you some time from writing reference values.

Loaders

loaders.callPackage

Source: src/loaders/callPackage.nix

Type: { self, super, root, ... } -> Path -> a

A wrapper around callPackageWith. It adds override and overrideDerivation to the output (as makeOverridable does), and requires the file being loaded to be a function that returns an attribute set. Unlike loaders.default, it will respect optional function arguments, as they can be overridden with the added override attribute.

loaders.default

Source: src/loaders/default.nix

Type: { self, super, root, ... } -> Path -> a

This is the default loader. It imports the file, and provides it the necessary inputs if the file is a function.

Default values of optional function arguments will be ignored, e.g. for { foo ? "bar" }: foo, "bar" will be ignored, and it requires inputs to contain foo. For that reason, although not strictly forbidden, optional arguments are discouraged since they are no-ops.

loaders.path

Source: src/loaders/path.nix

Type: { ... } -> Path -> Path

This loader will simply return the path of the file without importing it.

loaders.scoped

Source: src/loaders/scoped.nix

Type: { self, super, root, ... } -> Path -> a

This is like loaders.default, except it uses scopedImport instead of import. With this loader, you don't have to explicitly declare the inputs with a lambda, since scopedImport will take care of it as if the file being loaded is wrapped with with inputs;.

loaders.verbatim

Source: src/loaders/verbatim.nix

Type: { ... } -> Path -> a

This loader will simply import the file without providing any input. It is useful when the files being loaded are mostly functions that don't require any external input.

Matchers

Type: { matches : String -> Bool, loader : { self, super, root, ... } -> Path -> a }

Matchers allows non-Nix files to be loaded.

This is used for the loader option of load, which will find the first matcher where matches returns true, and use its loader to load the file.

matches takes the name of the file with (up to 2) extra preceding _s removed. For both bar.nix and foo/__bar.nix, the string matches gets will be bar.nix.

loader works exactly like passing a function to the loader option, the only difference is that the matcher interface allows loading non-Nix files.

When using matchers, the attribute name will be the file name without its extension, which will be foo for all of the following files:

  • foo.nix
  • bar/_foo.nix
  • baz/foo

Only the last file extension is removed, so far.bar.baz will have an attribute name of foo.bar.

matchers.always

Source: src/matchers/always.nix

Type: ({ self, super, root, ... } -> Path -> a }) -> Matcher

Matches any file name. This can be used as the last matcher as a catch-all.

matchers.extension

Source: src/matchers/extension.nix

Type: String -> ({ self, super, root, ... } -> Path -> a }) -> Matcher

Matches files with the given extension. matchers.extension "foo" matches a.foo and a.b.foo, but not .foo.

matchers.json

Source: src/matchers/json.nix

Type: Matcher

Matches all JSON files and loads them using lib.importJSON.

matchers.nix

Source: src/matchers/nix.nix

Type: ({ self, super, root, ... } -> Path -> a }) -> Matcher

Matches files that end in .nix. This is equivalent to matchers.extension "nix".

This is the default matcher if no matchers are defined.

matchers.regex

Source: src/matchers/regex.nix

Type: (regex : String) -> ([ String ] -> { self, super, root, ... } -> Path -> a }) -> Matcher

Matches the file name using the given regex. Instead of a regular loader, the function will also take the regex matches returned by builtins.match, as shown in the type signature ([ String ]).

matchers.toml

Source: src/matchers/toml.nix

Type: Matcher

Matches all TOML files and loads them using lib.importTOML.

Transformers

transformers.hoistAttrs

Source: src/transformers/hoistAttrs.nix

Type: (from : String) -> (to : String) -> [ String ] -> { ... } -> { ... }

This transformer will hoist any attribute of type Attrs with key ${from} up the chain. When the root node is reached, it will be renamed to an attribute of type Attrs with key ${to} and as such presented back to the consumer.

Neighbouring lists are concatenated (recursiveUpdate) during hoisting. Root doesn't concat ${from} declarations, use ${to} at the root.

This can be used to declare options locally at the leaves of the configuration tree, where the NixOS module system would not otherwise tolerate them.

transformers.hoistLists

Source: src/transformers/hoistLists.nix

Type: (from : String) -> (to : String) -> [ String ] -> { ... } -> { ... }

This transformer will hoist any attribute of type List with key ${from} up the chain. When the root node is reached, it will be renamed to an attribute of type List with key ${to} and as such presented back to the consumer.

Neighbouring lists are concatenated (++) during hoisting. Root doesn't concat ${from} declarations, use ${to} at the root.

This can be used to declare imports locally at the leaves of the configuration tree, where the NixOS module system would not otherwise tolerate them.

transformers.liftDefault

Source: src/transformers/liftDefault.nix

Type: [ String ] -> { ... } -> { ... }

This transformer will lift the contents of default into the module. It will fail if default is not an attribute set, or has any overlapping attributes with the module.

transformers.prependUnderscore

Source: src/transformers/prependUnderscore.nix

Type: [ String ] -> { ... } -> { ... }

This transformer prepends _ to attributes that are not valid identifiers, e.g. 42 -> _42. Attributes that are already valid identifiers (e.g. foo) are left unchanged.

Contributing to Haumea

Unless explicitly stated, all contributions are licensed under MPL-2.0, the license used by haumea.

Making Changes to the API

This doesn't apply to bug fixes.

  • Discuss before opening a pull request, so your work doesn't go to waste. Anything from GitHub issues to Matrix discussions is fine.
  • Update documentation accordingly. Everything in haumea.lib should be documented.
  • Add tests when necessary. Test your changes with nix flake check. Make sure new files are added to git.

Documentation

Documentation sits in the docs directory. You can get started with nix develop ./docs, which will start up mdbook and serve the documentation on localhost.

Scope

Haumea only depends on nixpkgs.lib. Features that depend on the rest of nixpkgs should not be added. However, changes that are specific to, but don't depend on nixpkgs are allowed.

Style

  • Format all Nix files with nixpkgs-fmt.
  • with should be avoided unless absolutely necessary, let inherit is preferred at almost all times.
  • rec attribute sets should be avoided at most times, use self or let-in instead.

See Also

  • namaka is a snapshot testing tool for Nix built on top of haumea with an interface similar to loadEvalTests.

  • paisano defines a directory structure for auto-importing. It is currently being reimplemented with haumea on the paisano-haumea branch.

  • std is a more full-featured framework based on the directory structure defined by paisano. Paisano was originally split out of std as a way to modularize std.

  • hive is a project built on top of haumea and paisano with a focus on building system and home configurations.

Changelog

v0.2.2 - 2023-05-26

Features

  • New book

  • load: loader now also accepts a list of matchers for loading non-Nix files (#10)

    The following matchers and functions available under matchers:

    • always always matches the file regardless of its file name
    • extension matches the file by its extension
    • json loads all JSON files
    • nix is the default matcher if the loader is a function and not a list of matchers
    • regex matches the file using the given regex
    • toml loads all TOML files

v0.2.1 - 2023-04-19

Features

  • loaders.scoped to utilize scopedImport instead of import for loading files

v0.2.0 - 2023-04-10

Breaking Changes

  • Transformers now accept a ccursor as an argument. The type signature of transformer have changed from { ... } -> a to [ String ] -> { ... } -> a

Features

  • transformers.hoistAttrs and transformers.hoistLists bring a specific attribute name at all levels to the root.
  • load: transformer now also accepts a list or a nested list of functions.

v0.1.1 - 2023-04-07

Features

  • load: add transformer option
  • transformers: add liftDefault

v0.1.0 - 2023-04-01

First release