System downtime caused by repeated copying of data

[sunxixixixi]

Bug Report: System Freeze During Sequential File Copy Operations on VMDK

Environment:

Host OS: Windows 10 22H2 (x64)
Virtual Disk: aim_cli mount VMDK image
cmd: aim_cli.exe --mount --filename=xx.vmdk --provider=DiscUtils --writable --background
Test File: ~900MB sample file
Reproduction Steps:

Mount the VMDK disk image
Add a test file (~900MB) to the mounted volume
Perform sequential copy operations with exponentially increasing file counts:
First iteration: Copy 1 file
Second iteration: Copy 2 files
Continue doubling (4, 8, 16, 32, 64...)
Observed Behavior:

I/O operations become unresponsive during the test
Complete system lockup occurs shortly after
Requires hard reset to recover

Issue appears consistently reproducible
Lockup occurs regardless of copy destination (same volume/different volume)
No BSOD or error messages generated before freeze

And shortly there after the entire system becomes unresponsive and needs to be hard reset.

[sunxixixixi]

The test file is 90MB in size, not 900MB. However, it seems that this does not affect the final result.

[LTRData]

Thanks for your report! I'll look into it!

[sunxixixixi]

Thank you for response,and I'd like to provide some more detailed investigation results
Additional Test Notes:
The project's data write mechanism is similar to how Windows handles VHD disks—both initially cache data in memory and then gradually write it to disk. However, in this project, there appears to be a flaw in flushing the cached data to disk, which caused the system to freeze. During the copy process, we observed that available memory decreased but did not recover, eventually leading to a system freeze.

[LTRData]

Thank you for response,and I'd like to provide some more detailed investigation results
Additional Test Notes:
The project's data write mechanism is similar to how Windows handles VHD disks—both initially cache data in memory and then gradually write it to disk. However, in this project, there appears to be a flaw in flushing the cached data to disk, which caused the system to freeze. During the copy process, we observed that available memory decreased but did not recover, eventually leading to a system freeze.

Yes you are absolutely right. I have been able to reproduce this in this particular Windows version and I can confirm after some debugging that the cache system gets stuck with very high load. It never recovers from it because flushing the vmdk file gets in the queue behind flushing the files to the virtual disk that depend on flushing this vmdk first etc.

I'll see what we can do about it.

Meanwhile, as an emergency recovery without hard reset, it seems in most cases possible to remove the virtual disk with aim_ll -d or aim_cli --dismount (without specific virtual disk number or mount point).

[sunxixixixi]

Regarding the memory flushing issue, it might also lead to another observation. After mounting the disk, I tried creating an empty file or folder and then restarting to test memory refresh efficiency.

Key Findings:

If the system restarts immediately after creating the file/folder, there's a relatively high chance that the remounted disk won’t show the newly created item.
This has minimal practical impact, as the issue is minor in real-world usage.
For VMDK format mounts, there seems to be a different memory refresh mechanism:
When copying a large file to a normal disk image, the image size gradually increases during the copy.
With VMDK, the size only updates after the copy completes.
If a restart happens mid-copy (depending on timing), data may not refresh properly.

If the initial memory crash issue is resolved, would that also help mitigate the problems described here—such as the file/folder disappearance after restart and the VMDK refresh behavior?

[LTRData]

No, that is another issue. If you do not manually dismount a virtual disk before reboot, there is no guarantee that buffers will be flushed in the correct order before the reboot. This is because there is no service involved, all image file I/O goes though a normal user process that is killed on logout before the virtual disk has flushed all cache.

[LTRData]

Not necessarily a complete fix but something to use for more experiments around the issue. With this change, it is possible to activate a lazy-write cache queue for writes to the image file. This way the write requests can be completed early on driver side and it no longer locks up the file system cache flush writers. This drains the cache queue quicker for the files that you write to the virtual disk but at the cost of taking up a potentially huge amount of memory temporarily on the image file side of it. It is not necessarily a good solution for all use cases, but it seems to help against this particular risk of locking up the file cache for the entire OS.

Use --lazywrite at command line to activate this extra cache layer. Or set UseLazyWrites property to true on the provider when mounting through API.

[LTRData]

There are a few more things that needs to be adjusted to make this safe enough for a public release. The lazy write queue needs to be thread safe in a few more ways etc.

It is also worth noticing that this new lazy-write feature makes it a lot more sensitive to unsafe dismounts. If you do not dismount with correct flush logic (such as aim_ll -d -m D: or similar) before shutdown, you will definitely get a corrupt vmdk image file or corrupt file system within it.

[sunxixixixi]

Ok. I will observe whether it causes other usage problems and the issues you mentioned. If my system crashes due to other problems, there will be no chance to call aim_cli --dismount at this time. If the disk image file is damaged, it will be a very serious fault for the data

[sunxixixixi]

It seems that the problem has not been alleviated much. Under concurrent copying, my system still crashes and requires a hard restart. Moreover, calling dismount at this time cannot unmount the disk.

[LTRData]

It seems that the problem has not been alleviated much. Under concurrent copying, my system still crashes and requires a hard restart. Moreover, calling dismount at this time cannot unmount the disk.

Same behavior, the only difference that you added --lazywrite to the command line? Or something else that you noticed somewhere?