NixOS 101 Course / Section 6: NixOS integration tests
Premium

6.4 Faster NixOS tests with systemd-nspawn containers

Lesson 25 of 28 · 7:23

placeholder thumbnail

Sign in, then buy this course to watch

Free account · no credit card needed

Video chapters

  1. 0:00 VM-based tests and full isolation
  2. 1:28 Why containers are faster and lighter
  3. 3:39 Container limitations and Nix daemon setup
Transcript

VM-based tests and full isolation

0:00 and the container feature. So this is a huge improvement that has been upstreamed into NixOS by us in like winter from 2025 to 2026 and that got us many many improvements. So first of all the thing is that the legacy mode would be like VMs. I wouldn't say legacy actually because sometimes you need VMs, right? If you need guest kernel or whatever that is different from your host then you need a VM and there are many legitimate tests for that. And here in that abbreviated test we simply say nodes and one and two and then creates two VMs and then they can ping each other for example. We have seen that already. The advantage of this here is that each VM gets its own kernel. We can also run like any window output that we like. So from the getty to desktop everything is possible because we are simulating a full virtual machine, right? And we can do there's basically no limitations. The only thing is that to have this running fast you need KVM on your system. So a cheap virtual machine that is a virtual machine itself, right? And then running virtual machines further is often not really the compatible choice for that. So it does work but it's then very often switching back to or falling back to software emulation and slow. On the other

Why containers are faster and lighter

1:31 side we have containers. Containers you can just instead of writing nodes you would be writing containers. And that's it. Suddenly your machine and one and two are containers. In this case it's systemd-nspawn containers. So systemd- nspawn is kind of a containerization mechanism that already comes pre installed with your systemd so to say. That's in my opinion much cooler than Docker for example which you have to install in addition to that and then they have limitations. systemd-nspawn containers literally run their own root process tree with their own systemd manager and the only thing that they don't run themselves is the kernel, right? So we are reusing the kernel and it's all kind of put into namespaces, file system namespace, isolation for processes and so on and so on. And each of them has their own network configuration. Sounds heavy but is very light and by that we get tests that are very fast because we are not booting a new system for every VM. It's also very CI friendly. The reason for that is that you can run these tests easily and cheaply on very cheap VMs that don't have KVM, right? So if your test doesn't need KVM because you don't need your own kernel or whatever then you can totally run with systemd-nspawn containers. What's also interesting is that the reason why this exists in the first place was that a customer asked us "hey can we have some kind of elegant way to run GPU tests with CUDA inside NixOS

3:07 kind of in the sandbox" and that's relatively difficult with VMs at least for sharing GPUs and with containers it's actually pretty simple. I will show you in another video how to do that too. And in general while this was built for GPUs we said okay let's upstream this because this will be great for the public and will enhance all kinds of tests and right now since that happened people rewrite many many tests to actually use containers instead of VMs and speed them up a lot. You have seen in that video where we have been playing

The full transcript comes with the lesson.

About this lesson

NixOS tests can run their machines as lightweight systemd-nspawn containers instead of VMs, a feature our team built and upstreamed into nixpkgs. This lesson compares both, shows when containers are the better choice, and explains the Nix daemon setup they need.

In this video

  • What VMs can do that containers can't, and the other way around
  • Why container tests are friendly to cheap CI machines without KVM
  • The GPU use case that started it all
  • Limitations to know about before you switch
  • The Nix daemon settings for container tests and for tests that mix containers and VMs

After this lesson

You can pick VMs or containers for each test and set up your build machines for both.