I just changed the port my Steam Deck’s SSH server listens on, masked a service and started and enabled another.
If you know most distros including the one SteamOS is based on, you know that these things are generally not possible without write access to the /etc directory. sshd_config which sets the port number resides there. So do user changes to systemd service files. Masking and enabling both count as changes in this context.
So I have disabled read-only mode, right? No. It’s there for a reason, and I know from experience that yes, it is possible to screw up packages on an Arch based system.
But hang on, haven’t we been told time and again that the OS mounts a read-only filesystem and you cannot make changes to it? Yes, but there is a but. And in hindsight it’s kind of a big, obvious but.
Did your Deck not ask you to name it when it started up first time? That requires write access to /etc/hostname. Did it not ask you to connect to wireless network? Using NetworkManager, as SteamOS does, that requires write access to /etc/NetworkManager/system-connections (if you want NM to remember the connection, that is). And so on.
So is /etc an exception to the rule? And wouldn’t that undermine the whole “protect the sanctity of the root filesystem at all costs” shtick? Yes and no, I think. Here’s the solution put in place by the SteamOS developers.
[ ~ ] mount | grep '/etc'
overlay on /etc type overlay (rw,relatime,lowerdir=/sysroot/etc,upperdir=/sysroot/var/lib/overlays/etc/upper,workdir=/sysroot/var/lib/overlays/etc/work)Now, I did not know what “overlay” meant here so I had to go look it up. Essentially, it is what it says. You take a “lower layer” directory (usually read-only) and combine it with an “upper layer” directory (usually with write access) and merge them, i.e. present them as one unified directory. If a subdirectory exists in both layers their contents are combined in the overlay mount.
What if a file exists in both layers?
When a name exists in both filesystems, the object in the ‘upper’ filesystem is visible while the object in the ‘lower’ filesystem is either hidden or, in the case of directories, merged with the ‘upper’ object.
https://www.kernel.org/doc/html/latest/filesystems/overlayfs.html
Neat. So any file I add (to the upper layer) is presented insted of the system one on the lower layer (if there is a double). What if I want to edit an existing file on the lower layer? Like sshd_config mentioned above?
When a file in the lower filesystem is accessed in a way the requires write-access, such as opening for write access, changing some metadata etc., the file is first copied from the lower filesystem to the upper filesystem (copy_up).
https://www.kernel.org/doc/html/latest/filesystems/overlayfs.html
So my upper layer obviously has a copy of sshd_config that differs by the “Port” line from the original on the lower layer.
OverlayFS also has options for deleting and renaming files and directories but that is definitely not where I am going with the Deck’s /etc directory. Small, reasonable adjustments, yes. Grand reorganisations, no.
Now, the million dollar question is: What happens if my upper layer additions mess things up? After all, it only takes a bad /etc/fstab to stop a boot dead in it’s tracks. There is an obvious solution. If boot with overlay fails, disable overlay and just use the original, lower layer. This would essentially amount to a factory reset on system settings.
Is this implemented? I don’t know. I am not sufficiently familiar with this technology to know where to check and I’m sure as heck not going to test it. If you do, please let us know in the comments.
