🥝 Daniël de Kok

Recent Notes

Nix Overlays

9 min read

In Nix, we often want to build a data structure where one value refers to another value within that same data structure. For example, consider this simplified package set:

pkgs =
  let
    final = {
      libxml2 = {
        pname = "libxml2";
        version = "2.13.8";
      };
      libxslt = {
        pname = "libxslt";
        version = "1.1.45";
        buildInputs = [ final.libxml2 ];
      };
    };
  in final

We use a let-binding so that we can refer to the data structure itself by the name final. This allows us to refer to the libxml attribute inside the libxslt.buildInputs by using final.libxml2.

Unfortunately, this breaks down when we want to update a value inside the data structure using the update (//) operator:

nix-repl> :p pkgs // { libxml2 = pkgs.libxml2 // { version = "2.13.9"; }; }
{
  libxml2 = {
    pname = "libxml2";
    version = "2.13.9";
  };
  libxslt = {
    buildInputs = [
      {
        pname = "libxml2";
        version = "2.13.8";
      }
    ];
    pname = "libxslt";
    version = "1.1.45";
  };
}

The top-level libxml2 attribute was updated, but the structure inside libxslt.buildInputs isn’t. Nix uses immutable data structures, so the update operator construct a new attrset that doesn’t preserve sharing that arises from self-references.

Before we continue, we will switch to a simpler self-referential data structure to keep the examples short:

attrset =
  let
    final = {
      a = 42;
      b = final.a;
    };
  in final

Making final really final

Overriding didn’t work as we expected in the previous example because final referred to the initial attrset, not the updated one. So how do we change final to refer to the final attribute set after any updates are applied?

The solution will seem a bit weird initially. Instead of directly constructing the attrset, we are going to make a function that takes the final attribute set and uses it to construct the attribute set. We are also going to call this function overlay1, since it will soon become clear that we are building our own first primitive version of overlays:

overlay1 = final: {
  a = 42;
  b = final.a;
};

Despite questions about how we are going to use this, it should look very logical; for the value of b, we are not using the value of a in the attribute set itself, but from the final attrset that someone provides through the argument final.

The seemingly weird part here is that final.a may actually be a from the attrset inside the function if we don’t update a. Suppose that we want to get out the set itself unmodified. How would we do this? We’d have to pass the result of the function to itself. This seems paradoxical, but because Nix is a lazily evaluated language, we can do this without any issues. Let’s call the result of calling this function x, then:

nix-repl> let x = overlay1 x; in x
{
  a = 42;
  b = 42;
}

We actually did it! We passed the result of the function to itself.

It turns out that we want to do this more often, so it is useful to turn this into a function. Let’s call it fix, because it is actually the fixed-point combinator:

fix = f:
  let
    x = f x;
  in x

Let’s try it on our little overlay function for verification:

nix-repl> fix overlay1
{
  a = 42;
  b = 42;
}

Good! Now let’s move on to how we could use this to do updates to the attrset.

Composing updates

We could try to support updating the attrset function using an attrset like { a = 84; }, which is certainly possible (and left as homework for the reader). But it is nicer if any updates could also refer to final, because we could do things like:

overlay2 = final: {
  a = 84;
  c = final.a * 2;
}

which should have the effect of updating a in the original attrset, while also introducing a new attribute c that is computed from the final a.

So, the question is now: how do we compose two functions overlay1 and overlay2? We can already establish the ingredients that we need:

  • final has to be passed to both functions.
  • The attrset produced by overlay2 must update the attrset produced by overlay1. For this we can use the update operator.

So, how do we get a final to pass to both functions? Simple! We again make a function that takes final as an argument: final: overlay1 final // overlay2 final. We can then use the fix combinator again:

nix-repl> fix (final: overlay1 final // overlay2 final)
{
  a = 84;
  b = 84;
  c = 168;
}

Since these kinds of compositions are something we often want to do, we should make a generic function out of it:

compose = f: g: final: f final // g final

So compose is a function that takes two functions that create an attrset and returns a function that takes final. Let’s see compose in action:

nix-repl> fix (compose overlay1 overlay2)
{
  a = 84;
  b = 84;
  c = 168;
}

In practice, we often want to combine more than two overlays, so then it’s nice to be able to provide a list of overlays. To this end, we define a new function composeList that applies compose backwards, so that e.g. composeList [ overlay1 overlay2 overlay3 ] is equivalent to compose overlay1 (compose overlay2 (compose overlay3 (final: {}))):

composeList = lib.foldr compose (final: {})

Note that the (final: {}) is used as the initialization of foldr. We could have also taken the last value of the list (overlay3), but this is shorter and more efficient. Let’s try it out:

nix-repl> fix (composeList [ overlay1 overlay2 (final: { c = "epic"; }) ])
{
  a = 84;
  b = 84;
  c = "epic";
}

Epic! We created our own little language for composing overlays. To recap:

fix = f:
  let
    x = f x;
  in x
compose = f: g: final: f final // g final
composeList = lib.foldr compose (final: {})

Values from the previous overlay

There is one feature missing from our implementation of overlays. There is no way to access attrsets from previous overlays. In practice this is handy if you want to override a derivation. The following does not work:

final: {
  # Epic fail: infinite recursion!
  libxml2 = final.libxml2.override {
    # ...
  };
}

This does not work, because to compute the final value of libxml2, we need to access the final value of libxml2. In other words, this leads to an infinite recursion. What we want instead is to have access to the previous definition of libxml2 to override it. To do so, we add a prev argument that should give us access to the definition provided by previous overlays:

final: prev: {
  libxml2 = prev.libxml2.override {
    # ...
  };
}

Let’s redefine some overlays with a prev argument to give access to the attrset defined by previous overlays:

overlay1 = final: prev: {
  a = 42;
}
 
overlay2 = final: prev: {
  a = 84;
  b = prev.a;
  c = final.a;
}

We will keep fix as-is:

fix = f:
  let
    x = f x;
  in x

The issue is though that fix expects a function with just one argument. We could solve this by converting the two-argument overlay to a single-argument overlay again by providing an empty prev to the (first) overlay. Let’s write a small function to do this:

emptyPrev = f: final: f final {}

Now fix works as expected:

nix-repl> fix (emptyPrev overlay1)
{ a = 42; }

Next, we have to update our compose function to work with the prev argument as well. Remember our original definition of compose:

composeOld = f: g: final: f final // g final

The wrinkle is now that we also need to both:

  • Use the result of f final prev in the update (//).
  • Provide the result of f final prev as the prev of g.

So, we will use a let-binding to compute f final prev once. Here is the new compose function:

compose = f: g: final: prev:
  let
    fApplied = f final prev;
  in
    fApplied // g final (prev // fApplied)

It may be a bit surprising that we also update prev with fApplied (prev // fApplied). The reason is that compose only combines attrsets produced by overlays (see the first // use). Since the initial prev does not come from an overlay, we need to make sure that it is also visible to g.

Let’s try our new composition function:

nix-repl> fix (emptyPrev (compose overlay1 overlay2))
{
  a = 84;
  b = 42;
  c = 84;
}

The only thing remaining is to update composeList to also handle the prev argument. The only thing we need to do is to ensure that the initial value has prev as well:

composeList = lib.foldr compose (final: prev: {})

Let’s try out the full set of functions with a list of overlays:

nix-repl> fix (emptyPrev (composeList [ overlay1 overlay2 (final: prev: { d = prev.a; }) ]))
{
  a = 84;
  b = 42;
  c = 84;
  d = 84;
}

To summarize, we now have:

fix = f:
  let
    x = f x;
  in x
 
compose = f: g: final: prev:
  let
    fApplied = f final prev;
  in
    fApplied // g final (prev // fApplied)
 
composeList = lib.foldr compose (final: prev: {})
 
emptyPrev = f: final: f final {}

If you made it this far, congrats! You can make your own overlay system in Nix!

Wrapping up

One thing that may still worry you is how overlays work in nixpkgs. In our overlay system, we always let the first overlay define the initial set of attributes. However, in nixpkgs, we apply the overlay to nixpkgs. Conceptually, this is very straightforward, for instance, if nixpkgs were bound to pkgs, we could make it the first overlay:

overlay0 = final: prev: pkgs

In practice, it is a bit more complex due to bootstrapping, but underneath, the same operators are used to do overlays in nixpkgs, only with slightly different naming:

Herenixpkgs
composecomposeExtensions
composeListcomposeManyExtensions
fixfix

nixpkgs also has some additional functions to make working with overlays easier and more powerful.

Overlays pop up a lot in and outside nixpkgs. For instance, the Hugging Face kernel builder uses overlays to build package sets with gradual refinement, such as in the ROCm and XPU package sets.

Backlinks