Planning to migrate a half-petabyte unRAID server to NonRAID

[tim000x3]

My Plan

So I'm about to do something that's either really clever or really stupid, and I figured I should introduce myself before I potentially become a cautionary tale in your issue tracker.

I'm planning to migrate my production unRAID server to NonRAID on Arch (btw). Half a petabyte. 28 disks. The whole thing. I've been doing extensive testing, already submitted a PR and an issue, and I wanted to share what I've learned and get some feedback before I pull the trigger.

My Current Setup (The one I'm Leaving Behind)

Running unRAID 7.1.4:

• 28-disk array (2 parity + 26 data)
• ~406 TB usable
• Mixed XFS and ZFS on the data drives
• 2 pool devices for cache
• Somewhere between 30 and 100 Docker containers depending on what I'm working on
• Drives ranging from 10TB to 24TB

It's been rock solid. But I'm a developer, I do a lot of open source work, and unRAID's locked-down environment has been driving me up a wall. I want a real Linux distro where I can actually manage things properly. Package management, AUR access, docker alternatives, the works.

But I'm not giving up parity protection on half a petabyte of data. That's where you come in.

The Testing

I wasn't about to just copy my superblock over and pray. I'm crazy, but I'm not THAT crazy.

So, I set up a bare-metal test system on Arch (btw) with kernel 6.12.1, NonRAID DKMS 1.3.2, kernel module 2.9.35, and beat on it with USB drives simulating a real array.

Test rig:

• Parity: 500GB HGST HTS725050A7E630
• Slot 1: 16GB Verbatim USB (XFS)
• Slot 2: 238GB SanDisk USB (ZFS)
• Replacement drive: second 16GB Verbatim

Results:

Test	Result
Array creation	PASS
Mixed filesystems (XFS + ZFS)	PASS
Parity sync	PASS
Hot remove / degraded mode	PASS
Read/write while degraded	PASS
Disk replacement workflow	PASS
Rebuild (~19 min for 14.7GB)	PASS
I/O during rebuild	PASS
Post-rebuild integrity	PASS
Reboot persistence	PASS*

Verified with SHA256 checksums at every stage. Everything matched.

What I Learned (The Hard Way)

The kernel module is solid. Almost every "bug" I found turned out to be PEBKAC. The module correctly detects failures, reconstructs from parity, handles replacement, rebuilds. All of it works.

Mount order matters. You HAVE to mount filesystems BEFORE starting a rebuild. If you start the rebuild first and then mount, writes update parity but not the already-rebuilt sectors. Stale data. Bad times.

The unassign step isn't optional. I initially thought the "Slot already has disk assigned" error was a bug. It's not. It's a safety feature. Full workflow is:

nmdctl unmount
nmdctl stop
nmdctl unassign SLOT
nmdctl replace SLOT
nmdctl start
nmdctl mount          # BEFORE rebuild
nmdctl check

USB drives are weird. Some USB drives report different vendor strings depending on cold boot vs hot-plug timing. Serial stays the same, but the prefix changes. Caused disk ID mismatches after reboot. This is a USB firmware quirk, not a NonRAID problem. SATA/SAS drives have stable IDs.

My Contributions

PR #91 documented the full disk replacement procedure, fixed the size display (was doing binary math but labeling as TB), and fixed the reload warning to include the unmount step.

Issue #92 found 13 prompts that ignore the -u/--unattended flag. The unassign command is the big one. Can't automate the replacement workflow without it.

The Migration

1. Boot Arch (btw) from USB with NonRAID installed via AUR
2. Copy my unRAID superblock to /nonraid.dat
3. nmdctl start and see what happens
4. If it works, install Arch (btw) to a real boot drive

I already tested loading my production superblock (without starting the array) and it parsed correctly. Disk IDs matched, slot assignments were right, sizes were right*.

*Once I realized it was calculating sizes in TiB but labeling as TB. Fixed in my PR.

Why I'm Posting This

Heads up. Someone's about to run this on a fairly large production system. If there's something I should know, now's a good time to mention it. (Or to get popcorn for the show, I guess)

Testing gaps I can fill. The README mentions some untested scenarios:

• Q parity rebuild (I have dual parity)
• Dual parity failure recovery
• Hot-swap under heavy I/O

Guess I will be testing these soon enough for better or worse.

Sanity check. Is my migration approach sound? Am I missing something obvious?

Long-term. I'm interested in contributing beyond just being a user. The kernel module seems solid. The rough edges are in nmdctl and documentation, which is exactly the kind of thing I like working on.

Let me know if you want more details on any of the testing, have suggestions, or just want to chat about data hoarding large-scale NonRAID setups. I'll be around.

[qvr]

Heh, cool! That would definitely be the largest setup for NonRAID I've heard about so far.

I'm biased, of course, but my assessment would also be that the driver part is solid, because it is the same thing that has been battle-tested a lot on unraid side. Where things are different, is of course nmdctl, but also the rest of the system setup which might cause surprising issues that unraid OS takes care of for their users.

For example the mount order thing you mention, that should have no impact on the driver, but as you mention using ZFS, if you have zfs-import-cache.service on Arch, and if you have ever imported the zfs pool manually, the service will remember the pool and try to directly mount the underlying raw disk device (on boot and possibly when reconnecting the USB device), and that will then cause parity corruption. My guess is this, or something similar happened to you, and the mount order during rebuild is not related.

Also regarding the unassign automation, if you simply want to automate it for testing purposes, that can already be done like I'm doing in the integration test workflow:
https://github.com/qvr/nonraid/blob/main/.github/workflows/dkms-integration-tests.yml#L230
I feel like having to do it in this kind of kludgey way makes sure a normal user won't accidentally shoot themselves in the foot with unattended disk unassigment.

[tim000x3]

Well, I'm about to start the migration in a couple hours. So wish me luck. I'll let you know how it goes.

I had a feeling you were gonna say something about not allowing unattended mode on a few things was intentional. That's why I opened that as an issue instead of making it part of my pull request.

Yeah, I mentioned over in the reply on the pull request that I never imported the pull manually. So it was super weird. But at least I did find a way to do it safely. Enough so that I feel comfortable just running this on the production machine now that I have a better understanding of it and the proper workflow. (Mostly just repeating myself here in case someone reads the discussion and not the pull request and has a similar issue)

[tim000x3]

The migration is complete. Everything runs normal. There are some things that are actually 300 times faster than unraid. I don't mean that as hyperbole as stupid as it sounds. Now time will tell for stability.

[phixxy]

What are some of the speed improvements you are experiencing?

I am curious to see how this goes for you. I think having a solid open source option is great, especially if something ever happens to limetech.

[FugginOld]

The migration is complete. Everything runs normal. There are some things that are actually 300 times faster than unraid. I don't mean that as hyperbole as stupid as it sounds. Now time will tell for stability.

Benchmarks or this comment will just be taken with a grain of salt.

[tim000x3]

The migration is complete. Everything runs normal. There are some things that are actually 300 times faster than unraid. I don't mean that as hyperbole as stupid as it sounds. Now time will tell for stability.

Benchmarks or this comment will just be taken with a grain of salt.

That's fair. Feel free to take it with a grain of salt. I didn't directly benchmark it. I knew that there was a process before that when I reboot my server took one hour to complete and it took just over 10 seconds. There are a few other things that used to take an hour and took a minute, which would be a 60x improvement.

Could there be other factors involved? Absolutely. However, it was never close to that fast before in my experience. Any time someone says 300 times faster, I generally take it with a grain of salt.

The problem is my unraid server wasn't very stable. It had a few issues so I haven't benchmarked it and I'm not gonna move back to unraid to run benchmarks. ( I sincerely apologize for the bad science) Unfortunately, I will admit because of this, I recognize that my reportings are "Trust me bro"

Sorry for the long reply. I just wanted to put this information out there.

[tim000x3]

What are some of the speed improvements you are experiencing?

I am curious to see how this goes for you. I think having a solid open source option is great, especially if something ever happens to limetech.

Absolutely anything that needs to scan the file system recursively, or do directory traversal, or crawl metadata.

[tim000x3]

I realized something I can show for speed would be the logs from my program

https://github.com/theweebcoders/CommercialBreaker

specifically

https://github.com/theweebcoders/CommercialBreaker/blob/main/ToonamiTools/utils/DirectoryScanner.py

Scanning anime library at /app/anime...
Fast scanning directory: /app/anime
Found 2714 show directories to scan
Progress: 100/2714 shows processed, 6933 files found in 39.9s
Progress: 200/2714 shows processed, 11118 files found in 72.4s
Progress: 300/2714 shows processed, 14496 files found in 83.7s
Progress: 400/2714 shows processed, 17250 files found in 91.9s
Progress: 500/2714 shows processed, 20233 files found in 92.7s
Progress: 600/2714 shows processed, 23357 files found in 100.5s
Progress: 700/2714 shows processed, 26285 files found in 113.8s
Progress: 800/2714 shows processed, 29881 files found in 131.0s
Progress: 900/2714 shows processed, 32894 files found in 166.7s
Progress: 1000/2714 shows processed, 35182 files found in 180.4s
Progress: 1100/2714 shows processed, 37533 files found in 185.5s
Progress: 1200/2714 shows processed, 40396 files found in 197.1s
Progress: 1300/2714 shows processed, 42726 files found in 207.1s
Progress: 1400/2714 shows processed, 45066 files found in 212.3s
Progress: 1500/2714 shows processed, 47226 files found in 217.1s
Progress: 1600/2714 shows processed, 49573 files found in 225.3s
Progress: 1700/2714 shows processed, 51617 files found in 232.5s
Progress: 1800/2714 shows processed, 53372 files found in 233.3s
Progress: 1900/2714 shows processed, 55344 files found in 234.2s
Progress: 2000/2714 shows processed, 57077 files found in 238.1s
Progress: 2100/2714 shows processed, 58048 files found in 239.0s
Progress: 2200/2714 shows processed, 59137 files found in 239.6s
Progress: 2300/2714 shows processed, 60212 files found in 240.2s
Progress: 2400/2714 shows processed, 61276 files found in 240.8s
Progress: 2500/2714 shows processed, 62190 files found in 241.6s
Progress: 2600/2714 shows processed, 63019 files found in 242.2s
Progress: 2700/2714 shows processed, 64135 files found in 243.3s
Finished scanning 2714 shows in 243.7 seconds
Processed 64261 files.

Unfortunately, I didn't save the logs from running this on unraid for comparison. However I can tell you, 263.6 files per second (to save you the math) is definitely an improvement.

To give a little more context, I'm also running 31 docker containers with at least 2-5 doing quite a bit of I/O operations of there own right now.

I know this isn't the most scientific of data, and for that I legitimately apologize, but I figured I'd at least post something.

[troyliu0105]

Very useful info for me to do some migrate test from unraid. 😄