Synology NAS Disaster Recovery

Good Sysadmin should do a good work especially for the worst things.

One of the things that usually new Sysadmin understand only after they already happens is Disaster Recovery.

Sooner or later, no matter how you work well or how much you pay for your hardware, something will fails or go wrong.
It may be a disk or a power supply, that's not so bad: if you are just something better that a young student you always use at least RAID1 and redundant power supply. Just hot/cold plug your spare part (because.. you have spare part right under your desk, don't you?) and you're done, back to your desk.

What if something worst happens? Don't thing about fire or earthquakes: if you have only one site and everything burs or fall down, well.. no one will work anyway even if you can restore your mail server in just half an hour!

Let's say that you have a NAS where you store all your server data.
What if it's motherboard fails?
Well.. I know that is quite hard that a CPU or chipset die nowadays but it can happen.
In this way you have all your redundant power supply and RAID5+hot spare disks that cannot be accessible..

The easy and costly solution is to have two of them: plug the disk from one to the other, restore the configuration and you're done: 10 min of downtime!
But it costs too much for something that probably will never happens in 10 years.

I face this problem a few days ago, when I decide to replace most of my storage with a centralised NAS. A SAN costs too much for my business, so my hardware vendor suggest me a good NAS: a Synology RackStation RS2212RP+.
That's a 10 SATA bay NAS, double Gigabit Ethernet with aggregation, redundant power supply and some really cool additional feature.
One of the coolest is that... it's Linux based!!!
What can you do with a Linux based business-line NAS without hardware RAID controller, in case of disaster recovery?
Just substitute it with a standard Linux PC until you receive the new NAS!!

It's pretty easy, if you know at least a bit of Linux administration via command line, to replace (at least for basic feature) this NAS. It will have a 0$ impact on your (always too small) IT budget (for sure you have a spare PC with 2-3 SATA port somethere.. don't throw it away too soon!!) and you can recovery your data in less than half an hour (if you are prepared!)

In my setup I have:

1. 4 HDD configured with RAID5 + Hot Spare. In this way 2 disk are enough to have a working RAID
2. Over this RAID5 I build a disk group, which allow me to create more that one iSCSI LUN and/or more than one NFS/SAMBA shares
3. I also create 1 LUN (and 1 iSCSI target) plus 1 volume shared via NFS  

This is a quite complex setup, and, probably, the most complex you can do with this Synology NAS. Less complex configuration (RAID1 or single volume instead disk group) will require less work.
This is what I've done when simulating a really bad hardware failure of my brand new NAS:

• plug 2 of the NAS disk (not the hot spare, of course!) in an empty Linux box. I used a standard Dell desktop, without any disk.

• put a Ubuntu 12.04 LTS Desktop USB dongle to boot from a Live distro

• once the boot is complete I need to get some more packages, which are not installed by default in the desktop edition:

sudo apt-get install mdadm lvm2

That's because I'll need to work with software RAID and LVM.

• Now, a bit of scanning to find the RAID device

mdadm --assemble --scan

The above command will scan physical disk to find an already created array. It will work automatically. With just 2 disk the array is degraded but data is accessible.

• It's time for LVM! Fortunately Synology guys didn't create something weird and custom but uses all Linux power to create the required flexibility of their "Disk Group" feature. Scanning for LVM is something like what I've done for RAID

1. sudo pvscan
2. sudo vgscan
3. sudo vgchange -a y 
4. sudo lvscan

Which

1. scan for physical volume (PV)
2. scan  PV to look for volume group (VG). This operation take a bit of time, but no more than a minute on my 2TB disks
3. enable finded VG
4. scan VG to look for logical volume (LV)

LV are the end of LVM stuff, if you are new to LVM let's say that it's like having a standard disk partition.
If you have just create "volumes" (speaking in Synology term) to export via NFS/samba, you're done. Just mount the LV and access your data!

sudo mount /dev/vg1/nfs_share /mnt/nfs

If you want to access iSCSI data you have two choice:

1. configure the iSCSI target on your Linux box to export LV and mount it on a client. This is a bit longer and it's outside the scope of this article. Take a look for example here for a really good tutorial.
2. mount the partition locally.

You cannot, AFAIK, mount the LV directly if it has been uses as iSCSI Target. In fact the client (initiator, in iSCSI terms) will see the Target as a disk, so it will, at least, build a partition table on it. So, before mount, you must scan for a partition table and create the right block devices. Nowadays this is pretty easy:

sudo kpartx -a -v /dev/vg1/iscsi_0

The above command will scan the LV and, if a partition table is found, create the corresponding devices. In my simple test-bed I had only one partition, so I can just mount it to have my files back on-line

sudo mount /dev/mapper/iscsi_0p1 /mnt/iscsi

That's all folks!
Now, order a new Synology NAS and replace the unlucky one!