As an IT guy, I have started to get a little bit of my own personal lab, and when sometimes something breaks, I always try to see it as a fun way to gain some experience and put on my troubleshooting hat. The fun stops however, when you know that you might possibly lost all of your personal data on your NAS .
This is what happened on my Synology DS415play after a power cut.
It started with the fact that my Synology was having disk errors on one of the disks and that the Synology itself was getting some weird problems. The Synology became sometimes unresponsive and became unreachable. For instance, ssh was rejected, shares couldn’t be accessed, the web interface was also down, and DSM assistant couldn’t even reach or see the Synology. A soft reset was not possible through one the interfaces, and the Synology also didn’t reboot by pushing the power button. So not an ideal situation. This all happened after an update, but I think this may also have been a problem in combination with the disk failure of one of the 4 disks that I had. I hoped that it was busy with some consistency checks, but after a few days letting it do its own thing, I noticed that the disks spinned up and down every 10 min by the sound that it was making. So the Synology seemed to be stuck in a sort of boot loop.
So stuck in a boot loop, with no possible way to view the web interface or console, I was getting pretty nervous. Luckily, I knew which disk was having problems, and so I decided to pull out the probably defected disk. I replaced the disk but still no improvement. Then, I decided to boot up without the faulty or replacement disk, and booted the Synology with only 3 disks. After that action Synology finally booted up, but the next panic attack started since I saw the screen above. Logic said that my data was still there, since I used a RAID 5 (SHR) setup with 4 disks, and only 1 disk was broken. I also knew that I removed the right disk, and that I didn’t touch the others. With my Synology up and running again, it was time to troubleshoot.
Getting my Synology volume back to work / mounted again.
Unfortunately, I’ve lost some of the screenshots that I made off the console, but I still know the commands and step that I’ve used. So first I connected to my Synology with ssh and checked if there was still a volume present. So I scanned for the Physical Volume (PV) with the command.
Syn-vSAM> lvm vgscan Reading all physical volumes. This may take a while... Found volume group "vg1000" using metadata type lvm2
So this was good news, It saw a volume group called “vg1000”. I Then tried to Enable this with the command.
Syn-vSAM> lvm vgchange -a y vg1000 1 logical volume(s) in volume group "vg1000" now active
So the Logical Volume in the volume group vg1000 became active.
A small step in the recovery process, but a big step for getting my hopes up.
After that I tried to mount the volume with the command.
Syn-vSAM> mount /dev/vg1000/lv /volume1 mount: mounting /dev/vg1000/lv on /volume1 failed: No such device
But unfortunately, this didn’t work. The next command gave some good insight about the logical structure of the PV, LV and VG.
Syn-vSAM> vgdisplay -v Finding all volume groups Finding volume group "vg1000" --- Volume group --- VG Name vg1000 System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 4 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 0 Max PV 0 Cur PV 1 Act PV 1 VG Size 5.90 TB PE Size 4.00 MB Total PE 1376322 Alloc PE / Size 1376322 / 5.90 TB Free PE / Size 0 / 0 VG UUID SSc872-duUFD-D8dbds…….. --- Logical volume --- LV Name /dev/vg1000/lv VG Name vg1000 LV UUID SSc872-duUFD-D8dbds…….. LV Write Access read/write LV Status available # open 0 LV Size 5.90 TB Current LE 1376322 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 4096 Block device 253:0 --- Physical volumes --- PV Name /dev/md2 PV UUID Sodi8-YHNCi-…. PV Status allocatable Total PE / Free PE 1376322/ 0
The data was still there as I could see that the Logical Volume Manager (LVM) still had the PV, LV, and VG mapped. So most probably there was some corruption that blocked the mounting of the volume.
Syn-vSAM> sudo cd /dev/vg1000/ Syn-vSAM> ls -la drwxr-xr-x 4 root root 45 Sep 22 15:35 . drwxr-xr-x 15 root root 13000 Sep 22 15:35 .. lrwxrwxrwx 1 root root 25 Sep 22 15:35 lv -> /dev/mapper/vg1000-lv
After that I checked the mapper.
Syn-vSAM> ls -la /dev/mapper/vg1000-lv brw------- 1 root root 253, 0 Sep 22 15:41 /dev/mapper/vg1000-lv
I noticed the 0, which after some googling, means 0 bytes. I actually found a similar post that was having a similar issue, which reconfirmed my thoughts. The Lv was corrupted.
So time to do a system file check.
Syn-vSAM> cat /etc/fstab none /proc proc defaults 0 0 /dev/root / ext4 defaults 1 1 /dev/vg1000/lv /volume1 ext4 usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,synoacl 0 0
With that I knew that the volume was ext4, and thus I could start the file system check with:
Syn-vSAM> fsck.ext4 /dev/vg1000/lv
After that I got a lot of (scary) messages which almost gave me the feeling that it was not recoverable. For a long time the Synology was checking the file system, and constantly prompting me if I wanted to take a corrective action for the corrupted data, which I replied with yes. After several hours, it was finally done.
I tried to mount the volume once more, but this time in read-only. Just to be sure.
Syn-vSAM> mount -o ro /dev/vg1000/lv
Afther that I listed the content with “ls” and I finally saw that it was mounted successfully.
Happy happy joy joy ^_^.
Then I rebooted the Synology so that it could mount the volume naturally. Once the Synology was rebooted, I had my volume back. The only thing left to do was rebuilding the RAID setup on a new replaced disk.
After that Everything was good 😊.
Samir is the author of vSAM.Pro and a Life enthusiast who works as a consultant in the field of IT. With a great passion for Technology & Personal Development, he loves to help people with their problems and challenges, but also inspire them with a positive and unique outlook on life.
Besides that, he is also a big Sport & Music junky that loves to spend a big chunk of his time on producing music or physically stretching himself.