Containers in NixOS
from [email protected] to [email protected] on 17 May 14:59
https://feddit.org/post/12610534

Hi everyone!

I’m in the process of finally doing containers right in my NixOS installation. This is my ‘wishlist’:

My current work-in-progress setup looks like this:

For each service (called $name), I have:

My containers are declared like this:

virtualisation.oci-containers.containers = {
    $name = {
        image = ...;
        ports = [ ... ];
        volumes = [
            "/srv/${name}/config:/config"
            ...
        ];
        user = $uid:$gid;
        extraOptions = [
            "--security-opt=no-new-privileges:true"
        ];
    };
};

Now for the parts I don’t fully understand yet:

Thanks for your input!

#selfhosted

threaded - newest

[email protected] on 17 May 15:33 next collapse

What services are you running in your pods/containers? Are they local applications like libreoffice or are they network accessible in the more traditional style? What’s the advantage to running a podman container on your machine vs a Flatpak container?

Sorry for all the questions. This is an interesting setup and I’m just really curious.

[email protected] on 17 May 16:53 next collapse

These containers are running on various servers I have at home, not on a desktop machine. I use podman as an alternative to docker, because it’s fully libre and does not require running containers as root. To be honest, I’ve never thought about running flatpak containers for these kinds of services – do you have a setup like this that you want to share?

[email protected] on 17 May 18:10 collapse

That makes sense. I’ve always thought of NixOS as a desktop distro, not as a server. Guess I need to expand my thinking!

I run Fedora Server with podman and docker side by side. I try to use podman whenever possible but sometimes it’s not worth the hassle so that’s when it becomes a docker container 😬

[email protected] on 17 May 20:48 collapse

I’d say NixOS is great for servers, mostly. Only having to worry about certain things (secure boot with custom keys, FDE, partition layout, network, sshd, firejail, etc.) once, and then replicating the same setup on another machine is waaay more convenient than going “I wonder what I was thinking when setting up this machine” once in a while when looking at some machine again you haven’t touched in some time. When it comes to desktop usage, the whole thing does not feel as magical - configuring system options in e.g. KDE is still a lot of clicking around in a GUI. I still use it for my desktop machine, just so I don’t have to think about another distro.

[deleted] on 17 May 20:47 collapse
.
[email protected] on 17 May 17:37 collapse

PUID is indeed handled inside the container itself, it'll run a container-provided script as whatever the container's UID 0 happens to be first which then drops to whatever $PUID happens to be inside the container. user= is enforced by Podman itself before the container starts, but Podman will still run as root in that setup. That means Podman is running "rootful", while if you started the container manually as $uid using the regular Podman CLI, it would be "rootless". That is a major difference in a lot of respects, including security, and you can find quite a bit of documentation on the differences between those operating modes online; it wouldn't fit in a comment. Rootless is generally considered the better mode, though there are some things that still require a rootful container.

In the upcoming NixOS 25.05 or current unstable, there are some tools you can use to run containers rootless as another user more easily using a new $name.podman.user = ""; setting. From what I understand they'll still be root-managed systemd system services that require sudo to operate, but that means privileges get dropped by systemd before running Podman, instead of dropped by Podman before running the container. This stuff is recent and I haven't used it, I just happen to know it exists, relevant nixpkgs commit if you wanna dig into it yourself: https://github.com/NixOS/nixpkgs/commit/7d443d378b07ad55686e9ba68faf16802c030025

[email protected] on 17 May 20:40 collapse

You just made my day, kind internet person! That’s exactly the holy grail setup I’ve been looking for for the last couple of months. Will try it out as soon as I can!