Can not relight scene correctly

[YessionCC]

I tried the relighting command, but no matter which light I used, the relighting results were the same incorrect image. just as follow:

[nexuslrf]

Hi, there might be multiple potential reasons:

1. the roughness scale in this scene does not match the roughness scale in the pre-trained light model. This is common because of roughness-light scale ambiguity and our neural representation may amplify this problem. The workaround is to tweak more values in --roughness_scale you can sweep this scale 0.1-2.0 to find visually good relighting results.
2. During the training, you may not properly initialize (and/or freeze) the weights in the color MLP. If so, you might not get a consistent lighting latent space for light swapping.

Good luck!

[PikaCat-OuO]

I figure out with github copilot that the code is somehow wrong.
You can see the WARN when doing relighting:
[WARN] missing keys: ['env_net.0.weight', 'env_net.0.bias', 'env_net.1.weight', 'env_net.1.bias', 'env_net.2.weight', 'env_net.2.bias', 'env_net.3.weight', 'env_net.3.bias']
You can do a quick check by running:

python -c "import torch; ckpt=torch.load('exps/envs_all_11_unit_en/checkpoints/env_ckpts/env_net_10.pth'); print([k for k in ckpt['model'].keys() if 'env' in k])"

['env_net0.weight', 'env_net0.bias', 'env_net1.weight', 'env_net1.bias', 'env_net2.weight', 'env_net2.bias', 'env_net3.weight', 'env_net3.bias']

python -c "import torch; ckpt=torch.load('exps/scenes/toaster/checkpoints/ngp_ep2000.pth'); print([k for k in ckpt['model'].keys() if 'env' in k])"

['env_net.0.weight', 'env_net.0.bias', 'env_net.1.weight', 'env_net.1.bias', 'env_net.2.weight', 'env_net.2.bias', 'env_net.3.weight', 'env_net.3.bias', 'renv_net.0.weight', 'renv_net.0.bias', 'renv_net.1.weight', 'renv_net.1.bias', 'renv_net.2.weight', 'renv_net.2.bias', 'renv_net.3.weight', 'renv_net.3.bias']

Clearly mismatch in weight names. You can fix it temporary with:

python -c "
import torch

def convert_env_checkpoint(input_path, output_path):
    checkpoint = torch.load(input_path)
    converted_state = {}
    
    for key, value in checkpoint['model'].items():
        if key.startswith('env_net') and not key.startswith('env_net.'):
            if key.startswith('env_net') and key[7:8].isdigit():
                new_key = key.replace('env_net', 'env_net.', 1)
                converted_state[new_key] = value
                print(f'Converting: {key} -> {new_key}')
            else:
                converted_state[key] = value
        else:
            converted_state[key] = value
    
    checkpoint['model'] = converted_state
    torch.save(checkpoint, output_path)
    print(f'Converted checkpoint saved to: {output_path}')

convert_env_checkpoint(
    'exps/envs_all_11_unit_en/checkpoints/env_ckpts/env_net_10.pth',
    'exps/envs_all_11_unit_en/checkpoints/env_ckpts/env_net_10_converted.pth'
)
"