I previously recounted an issue with my blog server wherein the MariaDB failed to start after a Fedora update. I found a work-around for that problem to get the blogs back up and running but the underlying issue remained: each time the server rebooted the MariaDB database would fail to start due to a missing /run/mariadb folder.
The root cause
My workaround involved manually re-creating the /run/mariadb folder each time the server restarted. This means I have to be aware of when the server might have gone through a restart cycle and I have to be in a position with my computer handy to perform the steps necessary to recreate the folder. This all makes it very likely that my blog will be in an outage state at least once a week and is simply kind of aggravating so, after another outage earlier this week, I decided to re-examine the situation.
Examining the process to create the /run files I identified the following elements:
- a folder called /usr/lib/tmpfiles.d: which is where all of the packages like MariaDB create their pre-run configurations (*.conf files)
- a folder called /etc/tmpfiles.d: where you can create override files for the directives in /usr/lib/tmpfiles.d
- the systemd-tmpfiles command: which parses the above files usually during service startup and creates temporary files according to the directives
I created a simplified version of mariadb.conf file in /etc/tmpfiles.d that only creates the /run/mariadb folder. I then attempted to execute the systemd-tmpfiles command against just that single configuration file and observed an error.
[root@kgadams kadams]# cat /etc/tmpfiles.d/mariadb.conf
d /run/mariadb 0755 mysql mysql -
[root@kgadams kadams]# systemd-tmpfiles --create /etc/tmpfiles.d/mariadb.conf
Detected unsafe path transition / (owned by 1026) ? /run (owned by root) during canonicalization of run.
[root@kgadams kadams]# cd /runI also observed the same error recurring hundreds of times if I use the systemd-tmpfiles command without a file parameter which causes it to execute all the configuration files in /usr/lib/tmpfiles.d (/etc/tmpfiles.d is the over-ride folder)
Based on a Google search I found a reference to the problem. It appears to be a security setting that prevents systemd-tmpfiles from working with folders that are owned by something other than root. This is intended to prevent privilege escalation techniques.
[root@kgadams run]# ls -ld /run
drwxr-xr-x. 34 root root 1060 Sep 2 09:28 /run
[root@kgadams run]# ls -ld /
drwxrwxrwx. 21 1026 users 4096 Aug 22 21:29 /
[root@kgadams run]# chown root:root /
[root@kgadams run]# ls -ld /
drwxrwxrwx. 21 root root 4096 Aug 22 21:29 /I don’t know why the root (/) folder was owned by user UID=1026 and group users, and the permissions look a bit too permissive. But changing the ownership was sufficient for systemd-tmpfiles to work. After making this change the /run/mariadb folder was created on restart and the MariaDB database started without error.
Summation
MariaDB has worked on my server for years without issue. The problem started with a Fedora update that occurred just before my previous server outage which occurred on (I believe) August 25. I have automatic upgrades enabled meaning a lot of things change fairly quickly on my server, so something occurred on that date that triggered the outage conditions. There was a kernel upgrade performed on my server on August 25 which causes a restart so it is possible that either the update itself or merely the reboot along with a prior change was the trigger.
That means that one of two things changed:
- The / directory had previously been owned by UID=1026 but the security rules were tightened
- The / directory ownership was changed by something in the Fedora update to UID=1026 instead of root
I don’t know how long the security rules associated with systemd-tmpfiles have been in place so I cannot eliminate #1 as a cause. As for #2: I have zero recollection of ever changing ownership of the / folder on my server. That doesn’t mean that I never did: the server has been operating for close to a decade so a lot can happen in that time.
My best guess is that the security rules changed and I am somehow at fault for the incorrect directory ownership. I suppose it is just barely possible that some Fedora package changed the ownership of the root folder, but at the moment I’m going to assume that isn’t the case. Regardless, I now count this problem as “Fixed” as my server can restart reliably without causing a database and, thereby, blog outage. Hurray!
