Pool usage does not appear to go down when files are deleted in the filesystem

[jbaublitz]

As reported by @bleve on IRC, there appears to be a problem with pool usage not going down when files are deleted. Even if usage goes down on the filesystem, the pool still reports a usage that is reflective of the original state of the filesystem usage. This could be a problem for things like monitoring so we should find a way to resolve this with minimal performance impacts.

Deleting snapshots does appear to cause the pool usage to go down. This leads me to believe that there is some disconnect between the removed files in the filesystem vs. deleting snapshots which is a thin pool operation.

The first thing I'll be investigating is fstrim and its effects on the thin pool using some different tunables for discard minimum extent length.

Related to stratis-storage/stratisd#2814

[jbaublitz]

@bmr-cymru Thanks so much for your pointers on discards! fstrim /mnt did not work but fstrim -m 1m /mnt did indeed work to reduce the usage! I'll discuss with the team to see how we can build this into stratisd to help users.

[bgurney-rh]

On running blktrace on the thin-pool device of a Stratis pool, I found that I wasn't seeing discards on the device until I reloaded it with the "no_discard_passdown" parameter removed. After that, I saw it decrease from "106685/266798" to "105661/266798", on a pool with a 260 GB solid state drive.

The above was after having written 120 GB of test data, and deleting a few 3 GB files. Notably, the "Total Physical" "used" reading in "stratis pool list" was at "120.88 GiB" until I rebooted, and then I saw it say "103.24 GiB". This might be a separate thing that's specific to stratisd.

[jbaublitz]

@bleve So far what I've determined:

• Stratis 3.0 will update the usage every 10 seconds.
• fstrim with default parameters will free up only available blocks in the thin pool. Our thin pool block size is 1m.
  • The thin pool will only free up multiples of the block size so if you have a number of small files, the thin pool may not be able to free up any space if part of the 1m block is still in use.
  • This seems more likely for a directory like /home.
• Mounting with -o discard may not free up space because the discards issued to the thin pool may be smaller than the 1m block size.

It seems like your best bet for this for now is to use Stratis 3.0 and use fstrim. Two notes:

• We want to provide a better solution for handling discards/reclaiming thin pool space in later versions.
• If you see that not very much space is being freed up, it may have to do with lots of small files spanning thin pool 1m blocks.

Does this help at all? Can you confirm this behavior on your end?

[jbaublitz]

@bleve One last note that may be helpful to you.

Before you continue, just be aware that we do not recommend modifying state outside of stratisd, so doing this will be at your own risk, but I still want to provide you with a workaround. If you set the pool low water mark to 0 periodically, this will force a refresh. The way to do this is a little bit complicated because it involves keeping all of the data table for the thin pool the exact same except for the low water mark. If you fail to do that, you will get into a very bad state.

The reason for this is that the pool usage is refreshed when the low water mark is triggered. stratis-storage/stratisd#2814 provides a resolution for this by periodically checking the status of the pool and filesystems in the background. However for the version you are using, this will only be triggered by devicemapper events. The low water mark is the only workable devicemapper event to use for this in the stratisd case. If you periodically set the low water mark to 0, this will force a low water mark to be triggered. stratisd will refresh the usage information whether it went up or down and then set the low water mark to 2GiB less than the current usage so it will be triggered again every 2GiB you write to the pool. However, as you might see, this does not work for the case where the usage goes down. The workaround will effectively trigger the behavior that we've added in stratis-storage/stratisd#2814.

One last note is that you may still need fstrim as I mentioned above given that even if you trigger a usage refresh, if that space has not been reclaimed by fstrim, the usage will not go down. Both are most likely needed.

Please let me know if this is helpful!

[bleve]

It's good to know. For now we have changed system fstrim.service ExecStart to:

/usr/sbin/fstrim --all --minimum 1m --verbose

To work around the issue.