Postmortems
buildbot.lix.systems out of free disk 2024-06-09
The buildbot box was returning "no free space" to basically any btrfs operation including collecting garbage. Yet df -h stated that it had disk around.
Damn it!!
https://ohthehugemanatee.org/blog/2019/02/11/btrfs-out-of-space-emergency-response/
The box has another disk on it that did have space, but it was ext4. So I did something inadvisable:
# the device we were going to migrate nix store to
mount /dev/sda1 /mnt
fallocate -l 20GiB /mnt/ohno
losetup -f /mnt/ohno
losetup -a
# bad evil!! do not do this! this is a great way how you break your fs if the machine goes down
btrfs device add /dev/loop0 /
btrfs balance start -dusage=10 /
nix-collect-garbage # for a bit, then ctrl-c'd
btrfs device delete /dev/loop0 /
This freed enough disk that the machine was unstuck. I then ran more of a garbage collection, which freed enough space to further recover the machine.