Crashed Synology Volume, and how to Restore (DS415 Play)

Intro

As an IT guy, I have started to get a little bit of my own personal lab, and when sometimes something breaks, I always try to see it as a fun way to gain some experience and put on my troubleshooting hat. The fun stops however, when you know that you might possibly lost all of your personal data on your NAS .

Background

This is what happened on my Synology DS415play after a power cut.

It started with the fact that my Synology was having disk errors on one of the disks and that the Synology itself was getting some weird problems. The Synology became sometimes unresponsive and became unreachable. For instance, ssh was rejected, shares couldn’t be accessed, the web interface was also down, and DSM assistant couldn’t even reach or see the Synology. A soft reset was not possible through one the interfaces, and the Synology also didn’t reboot by pushing the power button. So not an ideal situation. This all happened after an update, but I think this may also have been a problem in combination with the disk failure of one of the 4 disks that I had. I hoped that it was busy with some consistency checks, but after a few days letting it do its own thing, I noticed that the disks spinned up and down every 10 min by the sound that it was making. So the Synology seemed to be stuck in a sort of boot loop.

So stuck in a boot loop, with no possible way to view the web interface or console, I was getting pretty nervous. Luckily, I knew which disk was having problems, and so I decided to pull out the probably defected disk. I replaced the disk but still no improvement. Then, I decided to boot up without the faulty or replacement disk, and booted the Synology with only 3 disks. After that action Synology finally booted up, but the next panic attack started since I saw the screen above. Logic said that my data was still there, since I used a RAID 5 (SHR) setup with 4 disks, and only 1 disk was broken. I also knew that I removed the right disk, and that I didn’t touch the others. With my Synology up and running again, it was time to troubleshoot.

Getting my Synology volume back to work / mounted again.

Unfortunately, I’ve lost some of the screenshots that I made off the console, but I still know the commands and step that I’ve used. So first I connected to my Synology with ssh and checked if there was still a volume present. So I scanned for the Physical Volume (PV) with the command.

Syn-vSAM> lvm vgscan
Reading all physical volumes.  This may take a while...
Found volume group "vg1000" using metadata type lvm2

 So this was good news, It saw a volume group called “vg1000”. I Then tried to Enable this with the command.

Syn-vSAM> lvm vgchange -a y vg1000
  1 logical volume(s) in volume group "vg1000" now active

So the Logical Volume in the volume group vg1000 became active.
A small step in the recovery process, but a big step for getting my hopes up.
After that I tried to mount the volume with the command.

Syn-vSAM> mount /dev/vg1000/lv /volume1
mount: mounting /dev/vg1000/lv on /volume1 failed: No such device

But unfortunately, this didn’t work. The next command gave some good insight about the logical structure of the PV, LV and VG.

Syn-vSAM> vgdisplay -v
Finding all volume groups
Finding volume group "vg1000"
--- Volume group ---
VG Name               vg1000
System ID
Format                lvm2
Metadata Areas        1
Metadata Sequence No  4
VG Access             read/write
VG Status             resizable
MAX LV                0
Cur LV                1
Open LV               0
Max PV                0
Cur PV                1
Act PV                1
VG Size               5.90 TB
PE Size               4.00 MB
Total PE              1376322
Alloc PE / Size       1376322 / 5.90 TB
Free  PE / Size       0 / 0
VG UUID               SSc872-duUFD-D8dbds……..

  --- Logical volume ---
LV Name                /dev/vg1000/lv
VG Name                vg1000
LV UUID                SSc872-duUFD-D8dbds……..
LV Write Access        read/write
LV Status              available
# open                 0
LV Size                5.90 TB
Current LE             1376322
Segments               1
Allocation             inherit
Read ahead sectors     auto
- currently set to     4096
Block device           253:0

  --- Physical volumes ---
PV Name               /dev/md2
PV UUID               Sodi8-YHNCi-….
PV Status             allocatable
Total PE / Free PE    1376322/ 0

The data was still there as I could see that the Logical Volume Manager (LVM) still had the PV, LV, and VG mapped. So most probably there was some corruption that blocked the mounting of the volume.

Syn-vSAM> sudo cd /dev/vg1000/
Syn-vSAM> ls -la
drwxr-xr-x    4 root     root            45 Sep 22 15:35 .
drwxr-xr-x   15 root     root         13000 Sep 22 15:35 ..
lrwxrwxrwx    1 root     root            25 Sep 22 15:35 lv -> /dev/mapper/vg1000-lv

After that I checked the mapper.

Syn-vSAM> ls -la /dev/mapper/vg1000-lv
brw-------    1 root     root      253,   0 Sep 22 15:41 /dev/mapper/vg1000-lv

I noticed the 0, which after some googling, means 0 bytes. I actually found a similar post that was having a similar issue, which reconfirmed my thoughts. The Lv was corrupted.

So time to do a system file check.

Syn-vSAM> cat /etc/fstab
none /proc proc defaults 0 0
/dev/root / ext4 defaults 1 1
/dev/vg1000/lv /volume1 ext4 usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,synoacl 0 0

With that I knew that the volume was ext4, and thus I could start the file system check with:

Syn-vSAM> fsck.ext4 /dev/vg1000/lv

After that I got a lot of (scary) messages which almost gave me the feeling that it was not recoverable. For a long time the Synology was checking the file system, and constantly prompting me if I wanted to take a corrective action for the corrupted data, which I replied with yes. After several hours, it was finally done.

I tried to mount the volume once more, but this time in read-only. Just to be sure.

Syn-vSAM> mount -o ro /dev/vg1000/lv

Afther that I listed the content with “ls” and I finally saw that it was mounted successfully.

Happy happy joy joy ^_^.

Then I rebooted the Synology so that it could mount the volume naturally. Once the Synology was rebooted, I had my volume back. The only thing left to do was rebuilding the RAID setup on a new replaced disk.

After that Everything was good 😊.

↑↑ Follow me on my Socialz ↑↑ - Or - ↓↓ Care & Share ↓↓

6 thoughts on “Crashed Synology Volume, and how to Restore (DS415 Play)

  1. Thanks
    It helped me a lot !!

    just notice that I had to free the device mapper before fsck. If not, I cannot do fsck :
    /dev/vg1000/lv is in use.
    e2fsck: Cannot continue, aborting.

    So with :
    dmsetup info -c
    brw——- 1 root root 253, 0 Jun 15 13:36 /dev/dm-0
    brw——- 1 root root 253, 1 Jun 15 13:36 /dev/dm-1

    ls -la /sys/dev/block/253\:0/holders
    lrwxrwxrwx 1 root root 0 Jun 15 16:54 dm-1 -> ../../dm-1

    253 is Maj and 0 is Min block for me

    I can check which mapper I need to remove, and then :
    dmsetup remove /dev/dm-1

    Now the logical volume is ready to be checked:
    fsck.ext4 /dev/vg1000/lv

    1. @Laurent, thank you for this information. I didn’t encounter that problem, but thank you for sharing this with me. Will be really helpful for other people who have this issue, so thank you for this. Were you able to recover your data afterwards?

      Best Regards,
      Samir

      1. Thank you both, guys! That helped me too.

        1. Happy To hear ^_^

  2. Just checking in to say.. THANK YOU! I had a power outage and when the system came back up, my volume was crashed. You saved me. Time to get a UPS and cloud backup, thanks again!

    1. Hey Eric,
      Great to hear, happy that this post could help you and thank you for your feedback as well 😊.

      Don’t forget that you can also backup your data on an external HDD with Synology.
      I always keep an offline backup as well every month, in case my complete system maybe gets crashed or maybe get one of those infamous crypto lockers. It makes you less invulnerable when something happens to your Synology.

      Have a great day.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.