Provide I/O benchmarks compared to unraid. Establish a performance baseline for future driver updates

[TheLinuxGuy]

Hey, I stumbled across this project and honestly it feels like it could be the missing piece of the “golden” NAS puzzle (https://github.com/TheLinuxGuy/free-unraid). Big thanks for putting this out there and sharing it!

A couple thoughts/ideas I wanted to throw in for consideration:
• Would be awesome to see some baseline benchmarks. Like, running the current driver on the same hardware (disks, CPU, etc.) compared to Unraid. That way we’d have a solid baseline to measure future improvements against and also see how it stacks up to Unraid.

Also curious — does the same ~33% write penalty that Unraid has show up here too? (ref: https://www.reddit.com/r/unRAID/comments/1fvnmpn/comment/lqaupno/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button)

[qvr]

The performance compared to a "real" UnRAID array on the same hardware should be equal - this is the same kernel driver running the array after all.

Likewise the write penalty is the same, and as mentioned in the linked reddit discussion, the default write method is the read-modify-write cycle, but it can be changed to "turbo mode" ie. reconstruct write, where the write performance is better, as it doesn't need to do multiple seeks on the same disk, but as a downside all array disks must be spinning.

I can try to run some simple validation performance tests at some point, testing disk write speeds when part of array and then "raw", but I don't think I personally will be able to test this with UnRAID. Like I mention in the project readme, I haven't actually used or even installed UnRAID ever, and I don't feel comfortable doing these kind of tests with a trial license. All NonRAID development has been based on the open source code distributed inside the install packages, and public docs + forum posts.

For what it's worth, I've been running NonRAID "in production" in my own Ubuntu based diy NAS for a while now, without any issues and performance has been "what I expect" from spinning rust.

If you need more I/O performance, then you can do separate "write cache disks" (like UnRAID has also) with mergerfs, and have a mover script handling moving the data to array disks, like: https://github.com/monstermuffin/mergerfs-cache-mover (That's something that I'm interested in trying out, but don't currently have hardware for.)

[TheLinuxGuy]

@qvr thanks for the reply! OK based on your suggestion I am trying to run some experiments but I ran into an issue and need your help. I am trying to use linux-bcache (/dev/bcacheN layers) as the storage disks for the array - which I speculate may give us NVME speeds on the entire array without dealing with pesky scripts moving data with mergerfs.

I'm unable to create an array using /dev/bcache0p1 because it is not a physical disks being filtered by the

nonraid/tools/nmdctl

Lines 2659 to 2681 in 378d6f6

	list_available_devices() {
	local devices=()
	# Find all disk devices
	for dev in $(lsblk -n -d -I 8 -o path 2>/dev/null); do
	# Find partition, largest on the disk and must not be mounted
	local partition=$(find_partition "$dev")
	if [ -n "$partition" ]; then
	# Get disk id
	local disk_id=""
	disk_id=$(udevadm info --query=property --property=ID_SERIAL --value --name="$dev" 2>/dev/null)
	# Get disk size information
	local size_kb=$(get_disk_size_kb "$partition")
	local size_gb=0
	if [ -n "$size_kb" ] && [ "$size_kb" -gt 0 ]; then
	size_gb=$(format_kbytes "$size_kb" 0 0 "gb")
	fi
	# Add to available devices
	devices+=("$dev|$partition|$size_kb|$size_gb|$disk_id")
	fi
	done

- do you have any manual steps or commands I can try to circumvent the check? Super brownie points if you can give me a quick run down on things I need to know to set everything up manually without the CLI as I presume you may be doing more filtering or checks elsewhere I haven't discovered yet.

My public notes on the experiment (so far) TheLinuxGuy/free-unraid@ea59101

[foux]

bcachefs will definitely write to multiple drives at the same time. Bit it also have a mechanism for favoring fastest drives so I hope that will mitigate the issue.
I'll try and let you know.

[foux]

Well bcachefs is unfortunately clearly not adapted. The performances are horrible.

[TheLinuxGuy]

bcachefs is a dumpster fire and I would not trust it. The developer got himself kicked out of the kernel for not playing nice.

My reason for exploring nonraid + bcache was that bcache-linux is used by Google in production, code is checked into the kernel and performance benchmarks I have done with it achieve great performance, even better than lvmcache and has a lot of sysfs configuration capabilities to make bcache settings whatever you like it to be.

[foux]

Hum, I didn't even think of lvm+lvmcache on top of the array. Might be interesting

[foux]

@TheLinuxGuy for your architecture, why don't you create the bcache on the nmdxp1 device, instead of creating it on the sdx device and then adding it to the array.

[foux]

@qvr pvcreate doesn't recognize the nmdxp1 block device, so I had to create a partition table on it, and a partition (/dev/nmdxp1p1). I suppose that won't create an issue?

[qvr]

Probably not from the driver perspective, but if you ever want to access the data via the underlying disk, the disk will have a partition table, which will have a partition that has another partition table in it. Might at least confuse some tools but if you're not worried about that scenario then yeah, it'll probably work.

But also, pvcreate should support raw devices and not require partitions, but the raw device must not have any partition table already. Did you try wiping the nonraid device with something like dd if=/dev/zero of=/dev/nmdXp1 bs=512 count=1 before pvcreate?

[foux]

Yeah, I've wiped completely the /dev/nmdXp1, but got this error from pvcreate : Cannot use /dev/nmd3p1: device type is unknown.
ANd indeed having the partition table inside the partition table is far from ideal :/

[foux]

I think adding "types = [ "nmd", 127 ]" in lvm.conf might help. I'll try that

[foux]

COnfirmed adding "types = [ "nmd", 127 ]" works. But we also have to put filter = [ "r|/dev/sd.*|" ] so that lvm won't scan sd devices before the nmd ones