General guideline on using AWS FSx Netapp Ontap on build machine

[narek-ham]

Hi All

@henrykie

Is there a recommended way to configure FSx Netapp Ontap for UE builds?

Inspired by

Building Games with Unreal Engine and Horde on AWS | Unreal Fest 2024

We are trying to use flexclone volume (also flex volume) via NFS SMB and getting strange, slow behavior plus RAM issues in build log though we get max 50-70% RAM usage.

We have tried to go up in resources up to 15000 IOPS and 512mb/s but it did not help at all.

Basically we are unable to build the project.

Editor target build via Build.bat:

Using Visual Studio 2022 14.38.33145 toolchain (C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.38.33130) and Windows 10.0.22621.0 SDK (C:\Program Files (x86)\Windows Kits\10).
Determining max actions to execute in parallel (32 physical cores, 32 logical cores)
  Executing up to 32 processes, one per physical core
Using Unreal Build Accelerator local executor to run 422 action(s)
  Storage capacity 40Gb
---- Starting trace: 250505_123747 ----
UbaSessionServer - Disable remote execution (remote sessions will finish current processes)
------ Building 422 action(s) started ------
UbaSessionServer - Allocation failed in VirtualAlloc (The paging file is too small for this operation to complete.).. process will sleep and try again
UbaSessionServer - Allocation failed in VirtualAlloc (The paging file is too small for this operation to complete.).. process will sleep and try again
UbaSessionServer - Allocation failed in VirtualAlloc (The paging file is too small for this operation to complete.).. process will sleep and try again

Building dedicated client or server via runUAT.bat:

    09:08:22  UbaSessionServer - Disable remote execution (remote sessions will finish current processes)
    09:08:22  ------ Building 4042 action(s) started ------
    09:08:28  UbaSessionServer - Allocation failed in MapViewOfFile (The paging file is too small for this operation to complete.).. process will sleep and try again
    09:08:28  [1/4042] Compile [x64] Module.EditorSubsystem.cpp
    09:08:28  [2/4042] Link [x64] UnrealEditor-EditorSubsystem.lib
    09:08:28  UbaSessionServer - Allocation failed in VirtualAlloc (The paging file is too small for this operation to complete.).. process will sleep and try again

    [108/4643] Copy [x86-64] ChaosClothingSimulationMesh.ispc.generated.h 
    UbaSessionServer - NtCreateFile for TraverseDir on \172.31.49.86\vol1_clone1 failed with error code c0000033 
    UbaSessionServer - NtCreateFile for TraverseDir on \172.31.49.86 failed with error code c0000033 
    UbaSessionServer - NtCreateFile for TraverseDir on \ failed with error code c0000033

Mup errors in Windows Event Viewer:

{Delayed Write Failed} Windows was unable to save all the data for the file ;E:000000000005df76\172.31.49.86\vol1_clone1\j_work_zspace.vs\zSpace\v17\Solution.VC.db-shm; the data has been lost. This error may be caused by network connectivity issues. Please try to save this file elsewhere.

The solution in this post does not seem to work

Could you share the configuration for creating/mounting /using NFS volume for build purposes?

Thanks in advance

Best Regards

[ghost]

This is a great ask. Let me look into this.

It differs significantly depending on CI/CD system. Horde agents have a configurable in appsettings.json that is relevant for mounting NFS volumes, but it does not provide automation for configuring the mounted volume. The errors you've shared seem to be page sizing issues with the volume, not an issue with actually mounting them. Seems like the machine being used as a Horde agent does recognize the mounted FlexClone, but is unable to write to it.

We'll do some experimentation on our end, and if you are able to resolve this lets keep the discussion going.

Are you in the Game DevOps slack workspace? That would be a great place to pose this question as well.

[narek-ham]

Hi,

I'd like to provide a bit more context.

We're currently exploring Unreal Horde and experimenting with its core components. From what we understand, Horde doesn't handle the underlying server/machine orchestration, which aligns with our approach of gradually integrating it with our existing infrastructure.

In our case, an entirely ephemeral approach isn't justified—static build machines help significantly reduce build times in gaming pipelines. Your talk about FlexClone volumes gave us the idea to maintain a single parent volume and use multiple FlexClones (one per ephemeral build machine). These clones would host the project, Derived Data Cache (DDC), and a custom-built Unreal Engine, while the build machines themselves would use small EBS volumes for compilers, libraries, and other tools.

We believe that setting up Horde is not technically required to experiment with this architecture—though we are working on integrating other parts of it in parallel—and this could be a viable cost optimization strategy

Currently, we're testing FlexClone volumes with our non-Horde-based infrastructure. The volume is mounted as part of our pipeline, it's recognized, and the build process starts. While we wouldn't say it's "unable to write," something seems to go wrong during the process.

Our concerns increased after reading this from Epic's support:

"I can't speak to the AWS FSx NetApp ONTAP configuration, as this is not what we use internally. We have investigated it and encountered some performance challenges at scale with slow/cold starts."
(Full thread: Epic Support Discussion)

If the approach we're testing is outside the scope of what you recommend or support, please let us know.

Thanks again for your time and guidance.

Best regards

P.S. I'm not currently in the Game DevOps Slack workspace, but I'd love to join.

[hananstrul]

Hi @narek-ham, I am writing here to understand if you were able to manually mount one of the Flex Clones with a client at all?
Please update if the NetApp TR (https://www.netapp.com/media/10720-tr-4067.pdf) assisted. Also, please let me know if you would like specific assistance in addition to the toolkit development process to better support your project.

[narek-ham]

Hi @hananstrul
As mentioned above we have successfully mounted Flex Clone volume on our build machine using NFS SMB protocol manually and as part of our pipeline (we have also tested and succeeded with iSCSI protocol just in case, not planning to use it yet).

[narek-ham]

My apologies, gentlemen. We’ve actually succeeded in connecting via the SMB protocol, following the instructions in this AWS documentation, using Workgroup mode.

I’m a bit embarrassed by the earlier confusion — thank you for your patience.

[hananstrul]

Yes, NFS will outperform SMB by nature; however, this is not the only factor to consider.
Mixed workloads can happen in several ways, and each one has its advantages or disadvantages that you need to be aware of in the user permissions and control.
Please read more below, and let me know what your preference would be
https://docs.netapp.com/us-en/ontap/nfs-admin/decide-security-style-svm-flexvol-concept.html
https://docs.netapp.com/us-en/ontap/nfs-admin/security-styles-their-effects-concept.html

[narek-ham]

We believe that UNIX style would be the best. If NFS shows good performance results, all our clients will be NSF on windows and we have reasonable experience with UNIX/Linux administration and we are planning to use this solution only for dedicated builds. The only concern here is the 3rd point which states:

An application accessing the data uses a UNIX user as the service account.

Since our builds are triggered by the Windows Administrator user, we're unsure how that will map to a valid UNIX identity on the NetApp side.

Additionally, we couldn’t find the “Mixed” option under Volume Security Style when creating the filesystem via the AWS Console—it seems to be deprecated (please correct us if we're mistaken). Is it possible to modify the security style after creation using the NetApp CLI?

If you have recommendations regarding which security style would be best suited for our setup, we’d greatly appreciate it. Otherwise, we’ll start by testing with “UNIX” and switch to “Mixed” if needed.

[hananstrul]

I have an alternative recommendation that may better address this challenge. By utilizing iSCSI LUNs for data access, you can achieve superior performance compared to the other options. If you'd like, we’d be happy to schedule a meeting to explore available alternatives and support you in selecting the optimal solution for your specific requirements

[narek-ham]

@hananstrul
We’ve actually tried using iSCSI as well, but we believe we didn’t set it up correctly. We’ll re-test and share the results. Here’s a brief summary of what we attempted:

Initially, we mounted the volume via SMB and copied the necessary files. After encountering performance issues, we switched to iSCSI. While the iSCSI mount was successful, the drive appeared empty, even though we expected to see the previously copied files.

Could you please confirm if our understanding is correct:
If we mount the parent volume via iSCSI from the beginning, copy the required files, then create a FlexClone from it and mount the clone (on the same or a different Windows client via iSCSI), we should see all the files present on the volume?

P.S.
If this next attempt also fails, we would be happy to take you up on the offer to schedule a meeting and discuss the available alternatives in more detail. Thanks again for your support!

[hananstrul]

Hi @narek-ham, iSCSI is a block protocol, whereas SMB/NFS are file protocols.
When using a file protocol, you have a filesystem that is managed on the ONTAP side
When using a block protocol like iSCSI or NVMe/TCP, you receive raw blocks. If this is the first use, you will need to format them according to your desired file system, taking into account your OS type and preferences.
ONTAP views the LUN as if it were a file inside the volume, and from that perspective, any other file written to the volume level will not be visible to the client.

[narek-ham]

Regarding NFS

We ran into permission issues as it was expected.
We successfully mounted the volume on Windows via NFS.

• 1st try: NTFS security style. We were unable to write into the disk, so we quickly switched to UNIX security style,
• 2nd try: UNIX style: the issue was we were unable to modify newly created (or copied) files as they did not have "Write" permission for "Others", so manual chmod was required each time a new file appeared on the disk. File ownership:
  using git bash on windows machine:

Administrator@EC2AMAZ-TGI40L6 MINGW64 /z/UE5
$ ls -ln ./


total 1532
-rwxr-xrwx  1 4278190080 197121     294 Mar 11 18:23  Default.uprojectdirs*
drwxr-xrwx 16 4278190080 197121    4096 Jul 14 15:20  Engine/
drwxr-xrwx  2 4278190080 197121    4096 Jul 14 15:52  FeaturePacks/
-rwxr-xrwx  1 4278190080 197121     417 Mar 11 18:24  GenerateProjectFiles.bat*
...
Administrator@EC2AMAZ-TGI40L6 MINGW64 /z/UE5
$ ls -l ./
total 1532
-rwxr-xrwx  1 Unix_User+0 197121     294 Mar 11 18:23  Default.uprojectdirs*
drwxr-xrwx 16 Unix_User+0 197121    4096 Jul 14 15:20  Engine/
drwxr-xrwx  2 Unix_User+0 197121    4096 Jul 14 15:52  FeaturePacks/
-rwxr-xrwx  1 Unix_User+0 197121     417 Mar 11 18:24  GenerateProjectFiles.bat*

on Linux machine:

ubuntu@ip-172-31-39-238:/mnt/test/UE5$ ls -ln
total 1532
-rwxr-xrwx  1 0 1     294 Mar 11 18:23  Default.uprojectdirs
drwxr-xrwx 16 0 1    4096 Jul 14 15:20  Engine
drwxr-xrwx  2 0 1    4096 Jul 14 15:52  FeaturePacks
-rwxr-xrwx  1 0 1     417 Mar 11 18:24  GenerateProjectFiles.bat
...
ubuntu@ip-172-31-39-238:/mnt/test/UE5$ ls -l
total 1532
-rwxr-xrwx  1 root daemon     294 Mar 11 18:23  Default.uprojectdirs
drwxr-xrwx 16 root daemon    4096 Jul 14 15:20  Engine
drwxr-xrwx  2 root daemon    4096 Jul 14 15:52  FeaturePacks
-rwxr-xrwx  1 root daemon     417 Mar 11 18:24  GenerateProjectFiles.bat

We got errors during build process as well

ERROR: Exhausted all retries!
Failed to delete file 'Z:\UE5\Engine\Plugins\VirtualProduction\Takes\Binaries\Win64\UnrealEditor-TakesCore.pdb'
    Exception: Access to the path 'Z:\UE5\Engine\Plugins\VirtualProduction\Takes\Binaries\Win64\UnrealEditor-TakesCore.pdb' is denied.
Attempting to retry...
Failed to delete file 'Z:\UE5\Engine\Plugins\VirtualProduction\Takes\Binaries\Win64\UnrealEditor-TakesCore.pdb'
    Exception: Access to the path 'Z:\UE5\Engine\Plugins\VirtualProduction\Takes\Binaries\Win64\UnrealEditor-TakesCore.pdb' is denied.
Attempting to retry...
Failed to delete file 'Z:\UE5\Engine\Plugins\VirtualProduction\Takes\Binaries\Win64\UnrealEditor-TakesCore.pdb'
    Exception: Access to the path 'Z:\UE5\Engine\Plugins\VirtualProduction\Takes\Binaries\Win64\UnrealEditor-TakesCore.pdb' is denied.
ERROR: Exhausted all retries!
Failed to delete file 'Z:\UE5\Engine\Intermediate\Build\Win64\x64\UnrealEditor\Development\AIGraph\UnrealEditor-AIGraph.lib'
    Exception: Access to the path 'Z:\UE5\Engine\Intermediate\Build\Win64\x64\UnrealEditor\Development\AIGraph\UnrealEditor-AIGraph.lib' is denied.
Attempting to retry...
Failed to delete file 'Z:\UE5\Engine\Intermediate\Build\Win64\x64\UnrealEditor\Development\AIGraph\UnrealEditor-AIGraph.lib'
    Exception: Access to the path 'Z:\UE5\Engine\Intermediate\Build\Win64\x64\UnrealEditor\Development\AIGraph\UnrealEditor-AIGraph.lib' is denied.
Attempting to retry...
Failed to delete file 'Z:\UE5\Engine\Intermediate\Build\Win64\x64\UnrealEditor\Development\AIGraph\UnrealEditor-AIGraph.lib'
    Exception: Access to the path 'Z:\UE5\Engine\Intermediate\Build\Win64\x64\UnrealEditor\Development\AIGraph\UnrealEditor-AIGraph.lib' is denied.
ERROR: Exhausted all retries!
Failed to delete file 'Z:\UE5\Engine\Binaries\Win64\UnrealEditor-AIGraph.pdb'
    Exception: Access to the path 'Z:\UE5\Engine\Binaries\Win64\UnrealEditor-AIGraph.pdb' is denied.