[UNREAL] Runtime Subsystem Extensions

[PedroRauizBeamable]

Basically, we need a way of letting users replace modify/extend existing runtime systems with sufficient granularity --- this is because there are a lot of cases where people need to call microservices and sometimes they want to do that as part of one or more operations.

For some cases, they could write their own subsystem wrapping ours:

• Before matchmaking, call a C#MS to update latency-to-server-location data and/or verify that you are allowed into the queue or not.
• Before joining an event, call a C#MS to update some custom Storage data.
• So on...

However, certain types of advanced cases benefit a lot from a different form-factor.

• Content Preloading: Imagine you want to preload assets defined in content objects --- now, you could do this AFTER the SDK is initialized but that adds unnecessary latency. If the SDK allowed you to hook into the content download flow, you could configure an operation to kick off the async loading of assets as the rest of cotent are downloaded --- essentially gaining some startup speed.
• MFA: You can think of our existing MFA flow as an extension hook too --- we give you an object, you do arbitrary things and then calls something on this object to let us know you've finished and the result of what you did. The SDK then responds to the result with well-defined semantics.

---

There are essentially three extension cases in total:

1. Well-Known Flow Extensions: There are a lot of cases that are common enough that merit a direct hook that exposes nice semantics that the SDK can provide out of the box: MFA, Content Preloading and Region Ping Update for matchmaking are examples of these. We should make these all in the mold of MFA --- hard-coded implementations that defer to user for the details and expect them to notify the SDK of the semantically relevant outcome of their custom code.
2. Login Semantics Extensions: Users can already do this by implementing a UBeamRuntimeSubsystem themselves. Our docs can be made better here. This is Cpp-only today. We need to improve the ergonomics of this in BP.
3. Uncommon Use-Case Escape Hatch: Letting game-makers replace our system's implementations entirely. This is useful ONLY if we provided the utility functions that can compose our original implementation such that people can format a new implementation.

The first two items on that list, we have a pretty good grasp on how to do. The last one however, is what needs designing and will be the bulk of the work on this ticket.

Here's a checklist of things that must be true for each of these cases:

1. We should add a few of the known cases:
   1. Matchmaking Ping Computation: We add a UpdateRegionPings that takes a RegionPings object onto which you are supposed to write a TMap<FString, float> containing the pings and an optional stat name (which defaults to beam.mm.region_pings) --- this writes to the stat. We can extend our existing C#MS utilities that fetches all of these from lobby players concurrently and then finds the best region
   2. Content Preloading: ???????
2. We should improve the BP-compatibility of this extension type. We should think about ordering and how to allow people to inject things to run before or after the set of Runtime Subsystems we ship with at each point of the SDK login lifecycle.
3. The following must be true:
   1. The extensions MUST be implementable in BPs.
   2. The extensions MUST be able to add output sub-event pins to our existing Operation nodes.
   3. The extensions MUST be per-system and hand-written (no need to hyper-generalize).
   4. The extensions MUST allow for more than one active (they are concurrently executed in the case of multiples).
   5. This doesn't really need to be in a sample (yet) but we need the docs to have some snippets.

---

There is another motivation for this which is to reduce update costs.

The SDK's code is available and modifiable so game-makers can already do all of the things above. However, they need to modify the SDK source (and flag it) and then re-apply the changes on every update. While in practice this works pretty well, the friction this particular form-factor adds to the process is a bit too much for certain dev cultures and as such make them less likely to upgrade.

We'd like to provide a path to extending our subsystems that lives outside of direct source modification because of that --- these hooks are that path.

---

Finally, docs. We are adding two pages to the docs for this feature:

• Modifying the SDK Source: a page where we teach people how to go about making changes to the SDK's code safely and techniques they can employ to reduce the upgrade friction enough that it is mostly trivial.
• Extending the SDK's Runtime: a page where we teach people how to go about extending the SDK runtime in two ways: the current "Make a new RuntimeSubsystem"-way and this new "Modify the Implementation of existing RuntimeSubsystems"-way.

In both of these pages we will explicitly say that while game-makers modify or extending the SDK means we cannot guarantee things will just work, we highly recommend experienced teams to take ownership of their core features through these techniques and C#MSs in order to avoid relying entirely on our scheduled Roadmap which may not align with their game's Roadmap targets.

We also need to point tech-leads to these pages at the end of the "SDK Lifecycle" page.

[PedroRauizBeamable]

We did a whole bunch of experiments here and ended up with the following results:

• Uncommon Use-Case Escape Hatch is not worth the cost since people have source access. We will be writing documentation for how to do this correctly and safely as well as implications for when you upgrade the SDK if you have source modifications.
• Well-Known Flow Extensions: we experimented with a "Hook"-based generic solution but it sucked for various reasons (readability of our source was much worse which made the ability to modify it much worse, the overhead for making a hook was pretty high --- interface + UObject + logic for sometimes a simple block of code, etc...). So... we decided to:
  • For pre and post event hooks (as in I want to do things before/after an operation and fail it if the things fail), we simply improved ergonomics in Blueprint (in C++ it was already fine). This was made by adding dependency pins to every operation node. So that you can chain operation dependencies easily in blueprint (this saves 1+N nodes per dependency chain link).
  • For hooks that happen in the middle of another operation, we'll handle these on a case-by-case basis using either a UObject based hook (content manifest hook) OR a operation sub-event + 2nd call (MFA implementation).
• Login Semantics Extensions: this is mostly a docs problem as well as a "we can make implementing (not using) Operations in BP easier so we should do some things to improve it". We're still exploring stuff here, but... this is unlikely to need breaking changes so we can do this incrementally.

In short, this ticket was very useful in that it prevent us from doing a bunch of things that would overcomplicate the SDK for very little gain --- now, we just need some docs updates to explain how to Modify the SDK Source (source change and upgrade considerations), Extend the SDK's Runtime (create your own Beam Runtime Subsystems) and How to chain operation dependencies (add to Blueprint docs).