Erroneous "No space left on device"

[kurczynski]

I've created a Stratis pool using two 6 TB LUKS devices:

$ stratis blockdev
Pool Name        Device Node                            Physical Size   Tier
redundant-pool   /dev/mapper/stratis-backup-storage-0        5.46 TiB   Data
redundant-pool   /dev/mapper/stratis-backup-storage-1        5.46 TiB   Data

$ stratis pool
Name                                 Total Physical
redundant-pool   10.92 TiB / 1020.59 GiB / 9.92 TiB

With a filesystem on top of it:

$ stratis fs
Pool Name        Name             Used          Created             Device                                   UUID
redundant-pool   backup-storage   1020.50 GiB   May 01 2020 12:24   /stratis/redundant-pool/backup-storage   75a9f6ee5f614d88b35c11f43899bcf9

However, I'm not able to use more than about 1 TB of space on the filesystem:

$ fallocate -l 10G test.img
fallocate: fallocate failed: No space left on device

Are there any other steps that need to be taken to use all of the available storage?

I'm using Stratis 2.0.1 on Fedora 32.

[drckeefe]

kurczynski, would you mind providing the blkid ouput? Also, are there any errors for stratis in the logs?

[drckeefe]

@kurczynski
We discussed this issue in stand-up today and require more information about this issue.
Can you provide information about how this filesystem was filled (which tools or applications)?
I wonder if it is possible if fallocate sparsely filled the filesystem without writing any data to the
underlying storage. In this case, it might be possible to consume the space without Stratis
knowing the filesystem should be extended.

[kurczynski]

I've since formatted the disks to use a btrfs raid0, so I don't have the blkid info. But I did check the logs and there was nothing but INFO and DEBUG level messages. I can post that if it may be helpful, it's over 500 lines though.

Of course. I was using rsync to transfer files from my single 6 TB btrfs disk to the Stratis pool I created and it failed with a No space left on device error during transfer. I restarted it a couple of times but was met with the same error. I ended up trying fallocate just to see if maybe it was something to do with the way rsync prepared files before sending them but as I posted, it came back with the same error.

[drckeefe]

Thank you for the feedback, kurczynski.
From the information provided in this issue I'm adding a link to this bugzilla. https://bugzilla.redhat.com/show_bug.cgi?id=1687974

@mulkieran do we have another github issue related to filesystems filling before a grow can complete?

[jwtrhs]

Sorry to revive an old issue, but I believe I'm experiencing the same behavior as OP. I've got a 4TB HDD with an SSD cache configured as a storage pool with a single filesystem.

$ stratis blockdev
Pool Name   Device Node      Physical Size    Tier
nas         /dev/md0              3.64 TiB    Data
nas         /dev/nvme0n1p3      209.26 GiB   Cache

$ stratis fs
Pool Name   Name   Used          Created             Device                 UUID
nas         nas    1017.06 GiB   Feb 16 2022 02:06   /dev/stratis/nas/nas   f0b957b8-4d9a-41e0-9434-8ee823ca8561

I'd been using rsync as well to copy a couple of TB of data to the stratis storage pool from another HDD, and am getting No space left on device errors.

Also trying with fallocate gives the same error as above.

$ fallocate -l 10G /mnt/nas/test.img
fallocate: fallocate failed: No space left on device

I'm on version 2.4.2.

$ stratis --version
2.4.2

Most of the files being copied with rsync are quite large (10-40GB) if that helps.

Is there any way to manually increase the size of a filesystem? Thanks.

Edit:
Also found this in the logs after creating the filesystem:

Feb 16 02:06:40 home-server stratisd[11147]: [2022-02-16T02:06:40Z INFO  libstratis::dbus_api::pool::pool_2_0::methods] The following filesystems name: nas, UUID: f0b957b8-4d9a-41e0-9434-8ee823ca8561 were successfully created
Feb 16 02:06:40 home-server stratisd[11147]: thread 'stratis-wt-5' panicked at 'assertion failed: `(left == right)`
Feb 16 02:06:40 home-server stratisd[11147]:   left: `Device { major: 253, minor: 7 }`,
Feb 16 02:06:40 home-server stratisd[11147]:  right: `Device { major: 253, minor: 0 }`', src/engine/strat_engine/thinpool/thinpool.rs:516:9
Feb 16 02:06:40 home-server stratisd[11147]: stack backtrace:
Feb 16 02:06:40 home-server stratisd[11147]: note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[drckeefe]

Could be related to this issue: #391
@jwtrhs do you have the output of stratis pool list?

[jwtrhs]

Thanks for the quick reply.

stratis pool list
Name                      Total Physical   Properties                                   UUID
nas    3.64 TiB / 1018.23 GiB / 2.64 TiB       Ca,~Cr   7d237765-7f70-4d59-8da0-48c80b47f432

I've added the filesystem to my fstab now, and am getting a log from stratisd saying an attempt to mount has failed due to missing metadata, though it does seem to succeed afterwards. Not sure if that's relevant.

Feb 16 22:32:53 home-server systemd[1]: Started Stratis daemon.
Feb 16 22:32:53 home-server stratisd[304]: [2022-02-16T22:32:53Z DEBUG libstratis::engine::strat_engine::liminal::identify] Stratis block device with Stratis device description: Stratis pool UUID: "7d237765-7f70-4d59-8da0-48c80b47f432", Stratis device UUID: "c34a4ef3-6d03-4fb6-a7dd-3368cab42150", device number: "259:3", devnode: "/dev/nvme0n1p3" identified
Feb 16 22:32:53 home-server stratisd[304]: [2022-02-16T22:32:53Z INFO  libstratis::engine::strat_engine::liminal::device_info] Device information Stratis device description: Stratis pool UUID: "7d237765-7f70-4d59-8da0-48c80b47f432", Stratis device UUID: "c34a4ef3-6d03-4fb6-a7dd-3368cab42150", device number: "259:3", devnode: "/dev/nvme0n1p3" discovered and inserted into the set for its pool UUID
Feb 16 22:32:53 home-server stratisd[304]: [2022-02-16T22:32:53Z INFO  libstratis::engine::strat_engine::liminal::liminal] Attempt to set up pool failed, but it may be possible to set up the pool later, if the situation changes: No metadata found on devices associated with pool UUID 7d237765-7f70-4d59-8da0-48c80b47f432
Feb 16 22:32:53 home-server stratisd[304]: [2022-02-16T22:32:53Z DEBUG libstratis::engine::strat_engine::liminal::identify] Stratis block device with Stratis device description: Stratis pool UUID: "7d237765-7f70-4d59-8da0-48c80b47f432", Stratis device UUID: "13dbd992-7838-4775-b312-30f4f9a97718", device number: "9:0", devnode: "/dev/md0" identified
Feb 16 22:32:53 home-server stratisd[304]: [2022-02-16T22:32:53Z INFO  libstratis::engine::strat_engine::liminal::device_info] Device information Stratis device description: Stratis pool UUID: "7d237765-7f70-4d59-8da0-48c80b47f432", Stratis device UUID: "13dbd992-7838-4775-b312-30f4f9a97718", device number: "9:0", devnode: "/dev/md0" discovered and inserted into the set for its pool UUID
Feb 16 22:32:55 home-server stratisd[304]: [2022-02-16T22:32:55Z INFO  libstratis::engine::strat_engine::thinpool::thinpool] Status of thinpool device with "devicemapper name: stratis-1-private-7d2377657f704d598da048c80b47f432-thinpool-pool, device number: 254:6, device node: /dev/dm-6" changed from "none" to "rw"
Feb 16 22:32:55 home-server stratisd[304]: [2022-02-16T22:32:55Z INFO  libstratis::engine::strat_engine::liminal::liminal] Pool with name "nas" and UUID "7d237765-7f70-4d59-8da0-48c80b47f432" set up
Feb 16 22:33:22 home-server stratisd[304]: [2022-02-16T22:33:22Z DEBUG libstratis::stratis::run] 11: thread started
Feb 16 22:33:32 home-server stratisd[304]: [2022-02-16T22:33:32Z DEBUG libstratis::stratis::run] 0: thread finished