Ok, where to begin? I am at work, playing with my VM (Virtual Machine), which might I add is not a good idea. I had just spent 3 hours prepping for a maintenance in the next few days and I modified a LOT of files on my VM to do this… I will also note ahead of time, I also didn’t make a snapshot (number one rule when messing around with an important VM, make a snapshot).
Basically, I was looking at my hard drive space… now usually when you are dealing with VMs (in this case using VirtualBox on Window) you are not really worried about space, after all it’s not your Primary system, it’s a virtual one for playing around, however in my case, it’s used as a full-blown secondary system which I use just as much as my main machine. As my main machine is Windows-based, and I am a Linux SysAdmin, having a Linux-based VM is where you do most of your work.
So while I was looking at my disk space, I noticed I was getting a little low and I needed to do a cleanup, which was successful by the way. But I was looking at a way to maybe add a little space. Now Fedora uses LVM or Logical Volume Manager, and I had run “pvs” to see what I had for space on my volume group. I didn’t like the group name, it was “vm-sys” which is the default, but it causes the mounts to look weird:
|
1 2 3 4 |
┌──{root@Koster-FC-VM:/home/mkoster}─────{Tue Feb 20 17:17:16}─── └──{# pvs PV VG Fmt Attr PSize PFree /dev/sda2 vm-sys lvm2 a-- <72.24g 0 |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
┌──{root@Koster-FC-VM:/home/mkoster}─────{Tue Feb 20 17:17:18}─── └──{# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 3.9G 0 3.9G 0% /dev tmpfs 3.9G 0 3.9G 0% /dev/shm tmpfs 3.9G 1.7M 3.9G 1% /run tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup /dev/mapper/vm--sys-root 18G 8.0G 8.7G 48% / /dev/sdb1 50G 4.3G 43G 10% /storage /dev/sda1 976M 184M 726M 21% /boot /dev/mapper/vm--sys-var 14G 3.8G 9.1G 30% /var /dev/mapper/vm--sys-home 33G 1.5G 30G 5% /home /dev/mapper/vm--sys-tmp 3.9G 17M 3.6G 1% /tmp Shared 1.9T 416G 1.5T 23% /media/sf_Shared tmpfs 798M 12K 798M 1% /run/user/42 tmpfs 798M 28K 798M 1% /run/user/1000 |
As you can see, it put in an extra dash, for example, “/dev/mapper/vm–sys-home” (wordpress of course removes the extra dash…). So I decided to change my vg name to be simply sysvg (short for system volume group) so instead of “vm–sys-home“ it would be “sysvg-home“. Well, this is where everything went batshit crazy on me. Right after I ran the vgrename command, I rebooted… I didn’t change anything else (you are supposed to change your fstab file and your grub config). My system was in full crash mode… it would not boot, you couldn’t even boot into single user mode… it simply had no idea where to boot from.
I look it up online, it tells me to use the Fedora Live CD to go into rescue mode, this is where the title of this blog post comes into play. It tells you to boot up the CD using “boot: linux rescue” as the command. It gets to a certain point and tells me that it cannot boot as the root user is locked. At first I thought this was because I had changed the root password on my Fedora, but after reading up on the documentation, and testing it once my system was fixed (I will tell you how shortly), I realized, it has to be a bug… of course the Root system is locked, it’s a live CD, thher is no root password on a Live CD… It’s supposed to use the –force flag to bypass this. So I look it up online and I get this bug report form 2016 –> https://bugzilla.redhat.com/show_bug.cgi?id=1371740 It explains the exact issue I have, however, as Fedora 23 was at end of life… Meh, fuck it, why bother fixing the issue, it’s an end of life product… except the bug still exists in Fedora 27, and no one cared to fix it… I might put in a bug report later if I have time, just to reference the old ticket and tell ’em to fix their shit.
Ok, so the main method of fixing my issue is a bust due to no one fixing an earlier bug that currently still exists. It’s now an hour later and I am frantically searching the web to try and find a solution. I come across a forum article that gave me the solution –> https://ask.fedoraproject.org/en/question/48318/failed-boot-after-lvm-volume-group-change/#_=_ Seems I am not the only fool who screwed up his system this way.
The first response got me on my way. It explains that at boot time, editing the kernel line that contains the old volume group name and replacing it with the new will get your system to boot… possibly… Well in my case it got halfway. It got to the point where instead of booting, instead of spewing issues that it could not find the drive, it simply went into Emergency Mode and asked me for my root password (which I had forgotten which of my passwords I used, as I never use Root). After finally guessing the right password I managed to run the vgrename command again to rename it back the way it was.
I rebooted again with my fingers crossed, and somehow my system was back online… With a huge sigh of relief as I need those files I spent 3 hours on, and to revert back to my previous snapshot, on Fedora 25 (6 months ago) there would be too many changed to account for. So Iam glad I finally found a way to repair my system, no thanks to Fedora’s Rescue method being an epic FAIL!.
https://bugzilla.redhat.com/show_bug.cgi?id=1547278
Maybe this time they will fix it before the EOL date.
Another UPDATE worth noting:
I was able to get into a rescue mode in Fedora, albeit not via the CD. I ran a bunch of tests to prove there is an issue for the bug report I submitted. During that testing I was only able to enter the rescue mode is the following was true:
- The file system was intact
- There was a root password set
That being said, a rescue environment SHOULD work, regardless of if there is a root password set, AND it should not rely on the machine itself. I still could not boot to a proper Rescue environment from the CD, which is where there should be a successful rescue environment. And in many cases, people do not set a root password for their local system. So in this case, I stand by my title that “Fedora Rescue doesn’t seem to work”. Hopefully, they will get the message and fix it this time, and not leave it till its no longer a supported OS.