Decide on another Renderer

[Capital-Asterisk]

PBR? Lights? Actual materials? Actual render passes?

We currently use Magnum, which is a lightweight abstraction around OpenGL and Vulkan bundled with useful tools, not a rendering engine. Magnum is excellent if we want to write a paper on cool a new GPU rendering technique, but not when we just want to "put a cube here, put a light here."

We can search online for open source PBR shaders and everything else; however, there's so much more infrastructure required to tie everything together: shader compilation, supporting multiple devices, scene environment map, etc...

Do I (or most other contributors I presume) want to put an extra 2 years of effort into an already-solved problem that's not actually related to doing any space-related things better? Probably not.

Other Rendering Engines

A practical solution arises: yoink an existing (open source) rendering engine

OSP features full separation between scene stuff and rendering, which means it's really easy to write a new rendering backend. OSP may be considered a 'custom game engine' but the only issue is that GAME ENGINES AREN'T REAL! WAKE UP!

Looking around online, there are no open source 'rendering engines' out there on their own, only ones from existing game engines.

Atom Renderer

I first looked at the Atom renderer from O3DE. O3DE's platform support, massive codebase size, and high minimum requirements is a deal-breaker for me though.

Godot RenderingServer

Dylan recently brought Godot RenderingServer to my attention. This apparently contains all of Godot's rendering capability and "bypasses the Scene/Node system entirely." The Scene/Node system is the thing I complain about the most. This aside, Godot is fairly lightweight, has good hardware & platform support, and a great community with lots of people using and testing it. I'm also kind of surprised how easy it is to download the editor, run, and export a project. This looks like our best option.

Should Godot use OSP as a library or should OSP use Godot as a library?

Ideally we should use Godot as a library, so we have access to the main function, command line interface, and control over running things in general. Imagine having an option to run a dedicated multiplayer server.

The main challenge here is that Godot is not designed to be used as a library. This is possible (to do without too much additional work) but is still under development: godotengine/godot#90510 . There's also possible issues with how Godot relies on global singletons, so we can't have multiple instances or may have problems reopening it after its closed.

"OSP as a library" is a much easier route; this is already a feature I intended anyways. The most convenient way to go about this is likely through making a GDExtension with OSP in it.

Would OSP become a Godot project with this route?

A dedicated server with just OSP can be built without it. OSP won't 'rely' on Godot; we'd be writing glue code that bridges the two (again, game engines aren't real). Reconsidering, most of the UI can be in Godot instead of RmlUi; it's likely worth getting the Godot folks from earlier in the project back on board.

Build-related notes:

• Godot uses scons, which is not CMake
• It's possible to build Godot manually without scenes and nodes. It takes about 5 mins to build on my machine.

[octanejohn]

U can use cmake for gdextension only if you want see related
https://github.com/godot-jolt/godot-jolt/blob/master/docs/hacking.md
godot-jolt/godot-jolt#886

[Lamakaio]

I have thought about that issue quite a bit in the last few days, and may start experimenting with it soon on my end. I'm outlining my thoughts here to get some feedback. Feel free to tell me anything not in line with your vision of the project, I honestly have no strong feelings on any of this, just a lot of free time to make use of.

General thoughts

• OSP can be useful as a standalone space sim library. That includes physics, all the mesh, weld, and terrain code, orbital simulations, etc
• A way to keep testing OSP without Godot would be nice. That is, keeping the magnum rendering code and current scenarios in some way.
• The Godot rendering engine, called through the RenderingServer API, would be used for more advanced graphics.
• The Godot editor could be very useful to design everything but the "space-sim" part of the game.
  • UI
  • All the graphics polish
  • Scenarios and "gamification". I think actually being able to place meshes in a scene in an editor, can be a very powerful tool. That means keeping some part of the Godot scene and node system, but only as a static "setup things" tool, which OSP could simply read.

The proposal

Disclaimer : this is a very high-level proposal, and I don't know if everything (or anything really) here would work in practice. It is simply my best guess after reading through things a bit.

graph LR
A[OSP magnum] --> |Depends on| B[OSP core]
C[OSP GDextension] --> |Depends on| B
D[OSP Godot] --> |Depends on| C
D--> |Made in| E[Godot editor]
C--> |calls| F[Godot RenderingServer]

Loading

This is my idea of the project organisation. It uses 3 C++ repos / folders : OSP-CORE, OSP-magnum and OSP-GdExtension.

I added a fourth repo, OSP-Godot, which would be a Godot project using the GdExtension.

The repos contain the following :

• OSP-CORE contains the space-sim code (physics, orbits, terrain ...). It exposes an interface through which game objects / settings can be added and interacted with, as well as a generic "Display world" which can be read by the graphics library.
  Another option here would be to store callbacks for grpahics primitives, which would be provided by the caller. However, it seems less flexible to me.
• OSP-Magnum does two things :
  • The "Magnum" part reads the OSP-CORE display world and displays it.
  • It also contains the current scenarios (and testapp), that call OSP-CORE to add their game objects and setting to the simulation, then start the simulation.
• OSP-GdExtension does two things :
  • read the OSP-CORE display world, and uses the Godot RenderingServer interface to display them on screen directly (bypassing the scene and node system)
  • Provides an interface to create scenarios through the editor, probably through custom Nodes. (Like an OSPGameObject kind of thing).
• OSP-Godot : the actual KSP-like game. Things like UI, a menu, settings, credits, sound... Would be here. The rocket editor would probably also fit best here. It uses OSP-GdExtension-provided nodes instead of the stock ones for game objects.
  Ideally, OSP-GODOT communicates as little as possible with OSP-GdExtension, like on menu actions, but not on every frame.

Caveats

• Performance :
  • The proposal include a lot of interface layers. Information transfer between these needs to be fast and / or limited to not be a bottleneck. Ideally, we should minimize any interaction between the Godot scene / node system and the space sim.
  • We would still have the Scene / Node system running, though not for everything. I'm not sure how performance-hungry that is (Not very, according to Godot docs, but well)
  • Multithreading : Think the architecture to need as little locks as possible.
• Building : All the c++ code should be able to be built using cmake. A template for building a gdextension with cmake exists here : https://github.com/asmaloney/GDExtensionTemplate
• More code to maintain / interface stability. We would have two (or 3) projects depending on OSP-CORE, making it a bigger headache to do any significant refactor.
• everything I didn't think of yet.

[Capital-Asterisk]

@Lamakaio
Oh this is perfect. Pretty much clarified most things I had in mind, just the specific implementation details left out of course. Nice diagrams too, didn't know github supported those.

Repository stuff

• OSP-Magnum and OSP-Core should stay together within the same git repository, since Magnum is used as a math library as well. OSP-Magnum is pretty much just the testapp; build options can be used to make it optional, which can done right now.
• OSP-GdExtension can still technically be made part of the same git repo and be optional, though I kind of want some separation away from it. Maybe we can make it part of the same git repo for now and separate it when it somehow gets annoying.
• OSP-Godot goes into its own repo for sure.
• I feel like we'd somehow have to reconsider our use of submodules, and allow disabling specific ones or just not use submodules.

Less important naming nonsense: OSP-Core somewhat conflicts with the existing core directory. OSP or libOSP are already taken, so might as well use the whole name: OpenSpaceProgram or libOpenSpaceProgram. (stupid suggestion: use numerical contraction "o14m")

Important stuff

It sounds like you're thinking of a more component-by-component level of integration into Godot, where OSP-GdExtension is mostly just an interface that passes calls down from OSP-Godot?

I see OSP-GdExtension doing most of the work, managing the tasks, scenes, and universe much like the testapp right now.

Consider "Rocket editor" vs. "Flight scene". Both will need to draw vehicles. Most of the vehicle logic and physics stuff used by a flight scene has to be managed entirely by OSP and only rendered by Godot. But now consider that we might want to edit the vehicle within the flight scene after launch, or maybe consider that we probably want the exact same rendering logic to be shared between both the editor and flight scene. This hints that OSP-GdExtension would have to do most of the 'game logic' as well.

If we choose to write the Rocket editor within Godot using GDScript/C#, then we'd have to write a bunch of interface code to have fine-grained control over the OSP side (no need to make Nodes for each part/entity, just a bunch of get/set functions or commands/messages).

Having a simple behaviour like changing the material of a part when the cursor is hovering over it feels like it would be a pain to do.

---

Another option here would be to store callbacks for grpahics primitives, which would be provided by the caller. However, it seems less flexible to me.

The ECS/task-friendly way to go about this is to use dirty flags and queues. OSP-GdExtension can read data written by OSP-core to sync it with Godot stuff, in much the same way that the Magnum integrations work.

I'm aware I've created quite a complex creature, but it's built the way it is to achieve a strong amount of separation between scene and renderer stuff. About Meshes, for example:

Meshes within scenes are represented by a MeshId:

• Unique to each scene and not shared.
• Exist even if the window is closed and the renderer doesn't exist.
• Can be associated with a 'resource' using ACtxDrawingRes to know where the mesh is loaded from.
• Can still be loaded from different sources without assiociating with a resource (I used this for the planet terrain stuff I'm still working on).

Then there's MeshGlId:

• Managed by the Magnum integrations (in "src/testapp/sessions/magnum.cpp").
• Associated with a MeshId, created in within the "Compile Resource Meshes to GL" task.
• Associated with GPU buffers.
• Deleted when the window and renderer is closed, MeshId continues to exist in the background

I see OSP-GdExtension simply associating each MeshId with a Godot Mesh.

---

We would still have the Scene / Node system running, though not for everything. I'm not sure how performance-hungry that is (Not very, according to Godot docs, but well)

Nodes are fairly massive: https://github.com/godotengine/godot/blob/master/scene/main/node.h#L161

OSP ActiveEnts, for comparison, are an order of magnitude faster and simpler. They literally just use 1 bit on their own, 12 bytes to put in the scene graph hierarchy, and maybe 50 or so bytes within vectors here and there.

I can see Nodes being used for editor gizmos, UI, a nav ball, and whenever we're forced to. Otherwise, RenderingServer can still be accessed through GDScript/C#.

[alex-sherwin]

(stupid suggestion: use numerical contraction "o14m")

Nerd sniped

... o3s4p5 ?

[asterane]

Wanted to note a few open-source projects I found, which claim to be rendering engines.

• Panda3D: https://docs.panda3d.org/1.10/cpp/index
• Filament: https://github.com/google/filament (seems to be about PBR specifically)
• OGRE: https://www.ogre3d.org/about/features (their ogre-next project in particular)

To me, OGRE especially appears to fit the bill of an open source rendering engine without 'game engine' facilities.

I have not looked deeply into these, and if they've already been disqualified, no problem. I'm listing them here just in case they might inspire simpler / more desirable architectural schemes.

[Lamakaio]

@asterane
OGRE does look quite good if we actually want the "renderer only" approach. I do want to look at how Godot integration would work, but both approaches have merit.

@Capital-Asterisk
I agree with most of what you've written here.

I'm pretty convinced now that the GdExtension should handle most of the "game" logic now, and just be fed UI / input events from Godot, and call the RenderingServer for graphics.

Just as a side note, Jolt seems to have about the same math primitives as Magnum, but with extensive SIMD support. It is not anything short-term, but it could be nice eventually to remove a dependency from the "full game" side of the project, as well as avoid a layer of translation.

[alex-sherwin]

Maybe it would introduce too much burden but… slightly crazy idea, it could be beneficial to try and support two renderering options to ensure the core doesn’t make accidental choices that end up tightly coupling it to a particular one.

Maybe not forever, maybe the 2nd choice is only there to ensure the “don’t tightly couple” doctrine without any expectation of going very far with it.

But FWIW, and this speculation comes purely from my research and following trends but Godot seems like it could be a good choice. Lots of pros in my opinion, could court more help, and Godot has a lot of momentum that only seems to be growing and will (probably) keep improving rapidly.

[Capital-Asterisk]

@asterane

These look like some fairly nice options. Godot feels like it would be the most artist-friendly with its editor though. I envision a workflow where materials/shaders are made in the editor, using Nodes as a preview/prototype. The arrangement of Nodes can be replicated on the OSP C++ side.

I'll keep these in mind for other projects though; thanks!

@Lamakaio

One thing I forgot about is src/adera: this is where the fun stuff and game mechanics are intended to go. OSP-GdExtension might just use it as a library for OSP-but-not-Godot-specific gameplay stuff.

Reconsidering the rocket editor after a while, I'm still not really sure if we should be listening to buttons clicks and stuff from C++, vs writing an interface to list off available parts and modify the vehicle.

Jolt seems to have about the same math primitives as Magnum...

It usually isn't too much of a problem to have a dozen different implementations of Vector3. Assuming most math would be inlined anyways, this ends up as more of a nuisance only the development side that doesn't make a difference on the binaries.

Magnum (technically Corrade) does have this weird runtime CPU detection feature but I don't think it's used for math.

Magnum is also used for loading vehicles and meshes. I'm quite a fan of it's 'Trade' components. The library is very well written imo, and there's a bunch of other general-purpose 3d things it can do as well. I'm not planning on removing it any time soon.

@alex-sherwin

Oh, you're in luck. I take coupling very seriously. There is already no tight coupling to any of the Magnum OpenGL parts, achieved 2 years ago in #171. Given familiarity with the codebase, it's actually straightforward to add a new renderer the test scenarios can use.

It's very helpful to keep the Magnum renderer / testapp around to help develop and test specific components (universe, vehicle logic, physics engine integration, etc...) without needing to go through Godot. This works as the "2nd choice"