When Docker containerization was the new hotness, many old hands sysadmins were repulsed by it. Not that the benefits were lost on them, but many aspects of system troubleshooting and maintenance were quite wacky. A lot of trusty tools and mental models just had to be thrown away, with nothing, or cobbled together prototypes, to replace them.
“How do I inspect the processes running inside”?
“Which networks is the container connected to”?
“How do I inspect the filesystem”?
“How do I snapshot its disk as a reference for comparison, and roll back to it later?”
“Why the hell it messes with my iptables ruleset?”
“I have neat and tight firewalling ruleset, but this docker container still has unrestricted network access?”
“You can’t make systemd start in container. You have to reimplement all services startup.”
“Where do the system logs go?”
“Oh, my reimplementation of services startup is not writing the logs properly.”
In so many regards, virtual machines are more palatable. Disk is a file, most of the same tools and ideas apply as for physical machines. Snapshotting is a first class feature. Networking modes are still a stumbling block, but at least the terminology is stable and adhered to by many vendors.
So I was quite happy to learn libvirt and guestfs which lets you come close to Dockerfile concepts when programming your virtual machine building process.
And the tooling, libvirt and guestfs, I expect to stay available and stable for many years to come.
Still, there are some problems with generating VMs with guestfs. Simple things such as properly sizing the disk are a hit and miss (you have to know the disk layout; layout using LVM is even trickier?), introspection into filesystem requires arcane commands, and updating grub bootloader is a problem if guesfs is not compiled with “grub” module (which is the case on Fedora). Running critical commands before the VM is installed into libvirt and booted up with SSH available is fraught with peril - getting output is hard, getting it clear and separate from, say, previous commands output is not possible, getting it in realtime I don’t know how. Quite a lot of faff over irrelevant details which stop you from focusing on your goals.
Here’s the kind of disposable deployment automation I did with libvirt for decent.im project.
Here comes LXD and its new fork Incus. I have heard for a long time about LXD, that these are containers but different. I didn’t get to play with them, which I regret now: I would come further ahead if I used LXD and not guestfs tooling. LXD and Incus come with public image servers: a rich collection of distro builds, in form of containers and VMs. To be fair, guestfs also has a collection of VM images, but the selection is not nearly as wide.
It’s not that Docker containers don’t let you look at ip addr or ip route (once you install those tools which are usually not included), but LXD/Incus containers make that sort of system exploration more immediately available and natural.
There are actual practical reasons to say Incus/LXD containers are more standalone and autonomous than Docker-ish.
For example, Incus ubuntu/22.04 container actually runs its init process and stuff, runs DHCP client and acquires and renews its dynamic IPs via the protocol. cron is running, /var/log/ is well-populated and journald is also at work. You can just install sshd and start it as you used to, e.g. systemctl start sshd.
root@bridged:~# ps axf
PID TTY STAT TIME COMMAND
199 pts/1 Ss 0:00 bash
964 pts/1 R+ 0:00 \_ ps axf
1 ? Ss 0:01 /sbin/init
117 ? Ss 0:00 /lib/systemd/systemd-journald
154 ? Ss 0:00 /lib/systemd/systemd-udevd
159 ? Ss 0:00 /lib/systemd/systemd-networkd
165 ? Ss 0:00 /usr/sbin/cron -f -P
166 ? Ss 0:00 @dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
169 ? Ss 0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
170 ? Ssl 0:00 /usr/sbin/rsyslogd -n -iNONE
171 ? Ss 0:00 /lib/systemd/systemd-logind
174 ? Ss 0:00 /lib/systemd/systemd-resolved
179 pts/0 Ss+ 0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 vt220
797 ? Ss 0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
917 ? Ss 0:00 \_ sshd: root@pts/2
935 pts/2 Ss+ 0:00 \_ -bash
920 ? Ss 0:00 /lib/systemd/systemd --user
921 ? S 0:00 \_ (sd-pam)
Containers can be nested, this is how:
incus launch images:ubuntu/22.04 nesting -c security.nesting=true -c security.privileged=true
Process tree view in ps axf is satisfyingly correct wrt nesting:
77248 ? Ss 0:00 [lxc monitor] /var/lib/incus/containers nesting
77258 ? Ss 0:01 \_ /sbin/init
77395 ? S<s 0:00 \_ /lib/systemd/systemd-journald
77434 ? Ss 0:00 \_ /lib/systemd/systemd-networkd
77438 ? Ss 0:00 \_ /lib/systemd/systemd-resolved
77440 ? Ss 0:00 \_ /usr/sbin/cron -f -P
77441 ? Ss 0:00 \_ @dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
77443 ? Ss 0:00 \_ /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
77444 ? Ssl 0:00 \_ /usr/sbin/rsyslogd -n -iNONE
77445 ? Ss 0:00 \_ /lib/systemd/systemd-logind
77453 pts/0 Ss+ 0:00 \_ /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 vt220
78188 ? Ssl 0:10 \_ incusd --group incus --logfile /var/log/incus/incusd.log
78273 ? Ss 0:00 | \_ dnsmasq --keep-in-foreground --strict-order --bind-interfaces --except-interface=lo --pid-file= --no-ping --interface=incusbr0 --dhcp-rapid-commit --no-negcache --quiet-dhc
79094 ? S 0:00 | \_ /opt/incus/bin/incusd forkexec nesting /var/lib/incus/containers /var/log/incus/nesting/lxc.conf 0 0 0 -- env LANG=C.UTF-8 TERM=screen-256color PATH=/usr/local/sbin:/usr/l
79097 ? Ss+ 0:00 | \_ bash
78356 ? Ss 0:00 \_ [lxc monitor] /var/lib/incus/containers nesting
78366 ? Ss 0:00 \_ /sbin/init
78464 ? S<s 0:00 \_ /lib/systemd/systemd-journald
78498 ? Ss 0:00 \_ /lib/systemd/systemd-networkd
78524 ? Ss 0:00 \_ /lib/systemd/systemd-resolved
78526 ? Ss 0:00 \_ /usr/sbin/cron -f -P
78527 ? Ss 0:00 \_ @dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
78530 ? Ssl 0:00 \_ /usr/sbin/rsyslogd -n -iNONE
78536 pts/0 Ss+ 0:00 \_ /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 vt220
79130 ? Ss 0:00 \_ /lib/systemd/systemd-logind
With guestfs, if you are running a VM bridged with your LAN, then libvirt cannot tell you its IP address, at least it doesn’t provide a tool. Incus does:
incus list --format json ${VM_NAME} | jq --raw-output .[0].state.network.eth0.addresses[0].address > ${VM_NAME}.ipv4
incus list --format json ${VM_NAME} | jq --raw-output .[0].state.network.eth0.addresses[1].address > ${VM_NAME}.ipv6
Incus containers can have their own Wireguard or OpenVPN interfaces.
Same Incus commands work with VMs and containers, all the difference is --vm flag to incus launch command.