Moving 271TB from Synology to Unraid – An Epic Journey

Writing to Each Disk in the Array Simultaneously with Rclone and Successfully Restoring from Backblaze

After eight years of running a Synology DS3615xs with an expansion unit (24 disks total), I finally decided to move to Unraid and retire my trusty Synology. It has served me well. I’ve read countless posts on data migration and tried several methods, and I found the following to be the fastest.

Preface:

– New Unraid box is an i9-14900K custom built 4U server rack
– Connected with a 20G SFP to a Unifi Aggregation switch
– 5Gbps network connection
– SuperMicro 4U 44-bay JBOD SAS3 – CSE-847E2C-R1K23JBOD

Backing up the Data

So this is a home lab and isn’t something I wanted to dump thousands of dollars into buying more storage. Over the years, I had picked up several drives for the Synology that, because of btrfs and the RAID 5 configuration, I wasn’t able to fully use the space on the 22TB drives I bought because I started with 12TB drives.

After a lot of research, I decided to back-up most of the server to Backblaze. I had about 130TB of drives just kicking around, so I started by creating the server, moving all my dockers over, and building the first array without cache or parity.

I began the backup of the Synology on December 12th, and it completed on January 4 using Synology’s Cloud Sync, which was really easy to setup. I set the number of simultaneous file transfers to 20 and let it run. 142TB backed up in total.

I didn’t want to back-up the data that I could transfer on the network from the Synology directly to save costs. So I moved several TB equally to folders on the Synology on the same share (so instant move, no transfer time). I named them 1, 2, 3, 4, 5 to keep it simple.

Total Backblaze cost was: $441.00

Rclone on Unraid

Once I had things prepped and the backup to Backblaze completed, I wanted to write to each of the disks all at once. I found that though I had a 10G connection, moving files to /mnt/user/ only produced about 200 MB/s. With a combination of writing to all the disks, including the NVME that will eventually be my cache drive (but not used as a cache right now, direct writing), I was able to get about 450 MB/s sustained.

I installed the Rclone Plugin and configured both Backblaze and Local Disk storage.

For each of the disks, I ran:

rclone move loc:/mnt/remotes/rdata/1 loc:/mnt/disk1/data/media/tv -vv -P --transfers=10 --progress --multi-thread-streams=0

I found this to be the best approach. Rclone moves the files off the Synology, deleting them as it goes, and then reports stats on how long each transfer will take.

This ensures that we don’t copy the same file twice, and also keeps files somewhat logically organized by disk. I repeated the commands for each folder and each disk going up in increments of 1 each time.

Of course, my Synology with its gutless INTEL Core i3-4130 processor is the bottleneck. I’m sure others could get better performance out of newer hardware.

Restoring from Backblaze

So since I didn’t want to invest in additional drives, mostly because I was missing about 200TB from my old RAID configuration that I’m reclaiming by moving to xfs, this solution let me transfer a great deal of data over, kill the Synology, and move the disks over (as I write this, I’ll be completing this step tomorrow after the final transfers are complete).

It is possible that some of the backup and direct data transfer will overlap. I decided to pause the Backblaze backup when I was comfortable, but since I gained so much space moving some drives from my Synology in advance, I wanted to direct copy as many files as I could over my network to save egress fees from Backblaze.

Once the local copy from Synology finished and I had verified the BackBlaze back-up succeeded, I removed all the drives from my Synology and placed them in the new Unraid server, giving me 397 TB of storage total.

The Rclone Plugin has a web interface that allows you to easily configure your storage provider. It doesn’t have to be Backblaze, it supports many, many providers.

For this, I turned back to Rclone and ran the following:

rclone move bb:Syno/tv loc:/mnt/user0/data/media/tv -vv -P --transfers=20 --progress --multi-thread-streams=0 --ignore-existing

This will move files from Backblaze, deleting them as it runs to save storage charges as the files are moved over. It will skip any directory it finds already in the folder on the Unraid, again, to save costs on redownloading something I had already moved locally.

This process took about ~4 weeks to upload, so it will likely take about the same amount of time to download. I’ll update the post later, but well worth it because I didn’t have to buy 24 new drives to accomplish this. Yes, I know many will say “that’s a waste of money,” but even at Server Part Deal prices, I would have been spending $5,400 vs. just paying for the temporary backup.

There are many ways to handle migrations, but I thought I’d share my success story in the event that it helped someone else.