Functional render graph frontend

[windwhiterain]

Abstract

If we see this project as a compiler that generates the actual graphics program, the current render graph is more like a kind of intermediate language and its backend. Maybe a functional render graph that use "operator", "slot", "connection" that compiles to current render graph is useful as a choice of its frontend.

Analyse

With a functional programing view, let's write a graphics program:

fn graphics_program(t: Texture)->(swapchain: Texture,){
    let a = defualt(); // a can be a color_attachment, its information is backward inferred from swapchain
    let b = f(&a, &t); // f can be a sample texture pass
    let c = g(&a, &t); // g can be another sample texture pass for post process
    let d = h(&b, &c); // h can be a post process pass
    return (swapchain: d,); // give the post process result to swapchain 
}

which is filled with allocations.

Functional proggram can be compiled to an un-functional version that reduce allocation like:

fn graphics_program(swapchain: &mut Texture, t: &Texture){
    let mut a = swapchain.clone();
    f(swapchain, t); // swapchain equals to b now
    g(&mut a, t); // a equals to c now
    h(swapchain, &a); // swapchain equals to d now
}

where only one allocation exists.

The automation provided by the current render graph is more like a furthur optimization for the un-functional program. I think the functional program will be prefered for users considering code simplicity, robustness and procedural graph building.

Multiple unordered modify

An example: multiple operators try to modify one resource expecting no other modification ahead. Users have to clone that resource in need.

Write in functional form:

let fs = vec![f0, f1, f2, ...];

fn graphics_program(t: Texture)->(swapchain: Texture,){
    let a = defualt();
    let bs = map(&fs, |f|{f(&a, &t)});
    let d = h(&bs);
    return (swapchain: d,);   
}

If we want to write this program in un-functional form, a should be modified by one of the operators in fs, the left operators should be fed with a clone of a. Like:

 
let fs = vec![f0, f1, f2, ...];

fn graphics_program(swapchain: &mut Texture, t: &Texture){
    let mut as = map(&fs[1..], |_|{swapchain.clone()});
    fs[0](swapchain, t);
    map(zip(&fs[1..], &mut as), |f, a|{f(a)});
    h(swapchain, &as);   
}

Which is quite indirect and error prone.

Mixed modify and read

Furthur more for the previous example: there are both operators try to modify and operators try to read one resource.

In functional form, we don't care if an operator will modify or just read an resource.

fn graphics_program(t: Texture)->(swapchain: Texture,){
    let a = defualt();
    let b = f(&a, &t); // f(x, t) will modify x
    let c = g(&a, &t); // g(x, t) will read x
    let d = h(&b, &c);
    return (swapchain: d,);   
}

Write in un-functional form:

fn graphics_program(swapchain: &mut Texture, t: &Texture){
    let mut a = swapchain.clone();
    let b = g(&mut a, t);
    f(swapchain, t);
    h(swapchain, &b);   
}

Since g only read a, we must call g ahead of f to avoid allocating intermediate resource, which furthur burdening the user.

Proposal

A higher level render graph that write in functional form, compile to current render graph.

Details

• pass1: build the graph data structure from code or serialized data.
• pass2: forward and backward type inference to determine each intermediate resource's information (size, format) in the graph.
• pass3:
  • determine operator execution order based on dependencies and minimal intermidiate resource allocation.
    • dependencies are implied by writing and reading to the same resource, all readers depend on the writer.
  • determine resource groups that each contains intermediate resources that should actually using one resource, automatically clone resources.
• pass4: generate current render graph.

[attackgoat]

I think the functional program will be prefered for users considering code simplicity, robustness and procedural graph building.

This functionality could be built using Screen-13 as a dependency, but it is not the target audience I am building for. I would be happy to help either adapt Screen-13 to fit this use case or explain how it currently builds a dependency graph and how this could form the basis of a new project.

I have some questions:

• Who is the target audience of the proposal?
• Is there an example of the proposed graph being used in industry or other programs?
• What goals are most important? (performance, flexibility, portability to other platforms/mobile?)

With a full description of what these passes are actually doing I am sure the existing resource aliasing and pass re-ordering logic would support the g(..) + f(..) + h(..) passes. If not that's a bug that needs fixing.

Background

The name Screen-13 comes from the BASIC language syntax for entering a specific low-resolution, high (for the era) color mode:

' This is QBasic code
CLS
SCREEN 13
DRAW "R10D10L10U10"

https://dos.zone/qbasic-1991/

The above example is exactly the ethos I want this crate to deliver on. The command "DRAW R10 D10 ..." is a concrete list of instructions which draw a box. Other graphics programmers will have different boxes to draw, and this crate should support those uses without imposing restrictions on the type or way boxes are drawn.

OpenGL

Another inspiration is OpenGL.

Using the API it provides is widely regarded as easy and allows programmers to work on graphics algorithms and ignore things like memory caching. Unfortunately, like BASIC, OpenGL is increasingly unsupported and not appropriate for new projects in 2025.

[windwhiterain]

Who is the target audience of the proposal?

Artists or technical artists who want quick graph editing, testing, frame debuging.

They might be familiar with "shader graph" in Unity or "material blueprint" in Unreal. However, render graph may enable them fully control the renderer.

Is there an example of the proposed graph being used in industry or other programs?

Falcor: this is the ideal form of this proposal, but it's Nvidia's you know, D3D only.

Rafx: this hide the allocations for intermidiate resources for user, but exposes too much low level details.

Bevy: in bevy's render graph, there exists "Slot" and "Edge". However, their developer decided to deprecate them and use global resource access.

What goals are most important? (performance, flexibility, portability to other platforms/mobile?)

I'd say flexibility. Functional programing has proven to be suitable for "node editor" based development in many game engines and CAD engines, especially when it comes to subgraph, procedural graph editing, modular graph building. A functional API might be like:

• All operators has only immutable reference parameter. The internal operator can accept mutable as well as immutable parameters, but both are wraped into the latter one.
• Operators specify their input slots, output slots.
• Graph add operators in any order, connect their input slots and output slots.

With a full description of what these passes are actually doing I am sure the existing resource aliasing and pass re-ordering logic would support the g(..) + f(..) + h(..) passes.

If we directly use current render graph in functional program without another compilation, then graph.copy_xxx must be able to let the graph know it's a "copy" rather than some actual operation so that the graph will have enough imformation to optimize. From what I have checked, graph.copy_xxx internally create a pass to copy, i'm not sure if the graph know that is a copy pass.

Why I'm here

I'm a cs student nearing graduation and a graphics lover, trying to gather some utilities for my future indie game.

This project is really cool for implementing the render graph optimize algrithm mentioned in GDC2017 while providing such a clean, easy to use interface. I'd like to gather some views from people first to see if we need a rust, cross-platform Falcor.

[attackgoat]

Ah, okay, I think I get it now. I misunderstood the scope of what you're creating. The previous detail about cloning/copying makes sense for describing a graph where the input is some node which is connected to multiple places. Each of those places should logically read the original version of the resource, but in some cases that copy could be elided.

Tracking "copies" and determining if they may be ignored is not something I want to build in to Screen-13 because it executes exactly the requested operations. Opinionated changes impact performance and complexity.

Building this layer as a new crate and compiling down to a Screen-13 render graph would certainly be possible. It does sound like an open-ended project and might get in the way of releasing an indie game though. As a stand-alone product it could be really valuable.

One big decision is whether or not to include shader operations as part of the graph: does the graph define the body of shader programs or accept them as blobs? Obviously Unreal engine generates shader code from the graph, which is super cool. SPIR-V is not super complicated and there are good tools to do this.

For any project I recommend starting with a proof of concept and working forward from that point. To get things started the graph could always copy resources (so they appear to be immutable).

In any case Screen-13 is stable enough to build production things and I expect to support it for at least the next few years. Correctness keeps getting better and performance, while already "good enough" is improving continually. There will be some API changes but nothing beyond search-and-replace type upgrades to user code.

Also:

• Screen-13 doesn't support Xbox/PlayStation; only targets Apple/Android/Linux/Windows
• Falcor says it supports Vulkan and DX12
• I built a graph-based editor for noise expressions using Egui which might help