The GUI overwrites my sceneconfig.txt

[MarilynKeller]

Hi,

I created a copy of run_dtu.py to create the configuration files for my own dataset. among the changes, I specify a SceneRotation of (0,0,0), while for the DTU dataset it is set to (-120,0,0).

After running run_mydata.py, the configuration files all look ok, sceneconfig.txt contains SceneReferenceFrame#SceneRotation vec3 0.0 0.0 0.0. I then launch the GUI with ./Release/ProbeSDF, load my sceneconfig.txt. But the cameras appear tilted, and when I open sceneconfig.txt, I see it as been overwritten with SceneReferenceFrame#SceneRotation vec3 -120.000000 0.000000 0.000000. The whole file gets overwritten so the other fields too.

I could not identify at what point it gets overwritten as it does not seem consistent. Once I even managed to load the cameras without that rotation but idk how it happened.

My cameras are positioned at the corners of an axis aligned cuboid. Once loaded they appear rotated:

If I run python ./scripts/run_mydata.py ./Release pathto/mydata -run_ours then it works fine, the 120 deg rotation is not applied and the reconstruction seems to work. But then I can't visualize real time.

Any idea what is going on?

[btoussai]

All the parameters are stored in the params.txt file.
The sceneconfig.txt file contains a subset of these parameters that control scene-specific data (orientation, bounding box, voxel size, path to images and so on). The specific subset of parameters that are relevant to the sceneconfig file is hardcoded in src/sequence/ParamsConfigSubset.h.
Whenever you drag and drop the sceneconfig file in the UI or when you run the headless program, the sceneconfig file is read and its content will overwrite what is contained in the params.txt file.
When the UI or the headless program is closed, the reverse happens and the sceneconfig file will be updated (as indicated in the console).
However, simply loading the scene does not reload the sceneconfig file since it is assumed to be in sync with the params.txt file.
TLDR. Just do the drag and drop again to reload the sceneconfig file.

Note that you can change the rotation from the UI itself using the "Scene reference tab" menu at the top.
You can change the bounding box in the "Reconstruction volume" tab in the left panel.
With this, you should be able to find a good combination of scene parameters that you can write down in your python setup script.

Your images contain a large amount of distortion. I think you should undistort them beforehand since the distortion polynomial may not match with what is implemented in src/sequence/cuda/projection.cuh. Make sure your backgrounds / masks are also undistorted and that the distortion parameters are set to zero in the updated calibration file to agree with the new images.
Note: the "Remove distortion" checkbox is only to visualize what the images look like with and without the distortion polynomial. It's purely for visualization and it doesn't do anything else.

If you have background images, you can try to use the built-in segmentation to compute the visual hull initialization.
The built-in segmentation is finicky so I don't recommend using it unless the backgrounds are very clean (identical to the input image everywhere except at the location of the object). There are a bunch of parameters controlling how the segmentation is computed in the "segmentation" tab of the left panel. Its basically a binary decision computed at each pixel, then smoothed and averaged at multiple levels of details.
You can have a look here to understand what the "threshold" parameters are for.
"Expected Foreground pixel tiles" controls the maximum number of 8x8 pixel tiles that can be allocated for the sparse segmented image.
"Expected Border pixels" controls the maximum number of pixels that can be allocated to hold the silhouette pixels needed to compute the visual hull.
You can change values beyond the range of the sliders with ctrl+left click and typing whatever you want on the keyboard.
The typical workflow to check that the segmentation and visual hull is good is the following:

1. Load the scene and verify that the bounding box / scene reference frame is correctly setup.
2. • Check "Take selected camera viewpoint" on the left panel. This brings you to the 2D mode.
3. • Check "Draw border". This will display the set of boundary pixels once the segmentation is computed.
4. • Verify that the images and the backgrounds are correctly loaded using the "Display Mode" combo box. You can also use the "1" and "2" keys (not the numpad ones).
5. • Click "Compute Segmentation". You can try adjusting the the threshold and boundary parameters if it's not good. It's okay if some of the pixels haven't been removed since the visual hull will remove inconsistent pixels. The opposite is not true: pixels that have been removed by the segmentation cannot be added back by the visual hull. Use the "Selected camera" slider to check all the images. Here is what it looks like with the MVMannequin dataset:

Now, to compute the visual hull:

1. Go back to the 3D mode: uncheck "Take selected camera viewpoint"
2. Go to "Visual hull" tab.
3. Check "Render Points". All the 2D points that are considered to be in the foreground will be drawn in red.
4. Click "Raytracing on borders". The foreground points will be iteratively projected onto the silhouette borders of all the images to generate the visual hull as a point cloud. This should look like this:
5. You can uncheck "Render Points" if everything is correct and go back to the "General" tab.

If everything is okay, you can simply click "Prepare volume". This will apply the segmentation, compute the visual hull, create an image pyramid (can be inspected with the "Image level of detail" slider) and initialize the voxel grid.

If you don't want to use the visual hull (too few cameras), you can instead initialize with an ellipsoid but still use masks.
To do so, you have to check "Skip Visual Hull" before loading the scene (or equivalently write "Window#SkipVisualHull bool 1" in the sceneconfig file).

The last possibility is to use masks (see DTU setup).

[MarilynKeller]

Hi,

Thanks, these precision are really helpful, I managed to change the parameters in the Gui and I'll try to apply the undistortion.

As for the segmentation, I do have masks, but while DTU works well, in my case they don't load properly in the Gui, I don't know why. My masks are black and white PNG like in DTU.

How it shows in the gui

My original png mask (don`t mind the noise, it is a mask with values 0 and 255).

It seems like the mask is loaded with the wrong format somehow. But I could not see any difference between the DTU masks and mine. If I create a mask by painting black and white in gimp, same issue arises.

[btoussai]

Hi,
from what I can see, the image is loaded correctly but the mask isn't applied correctly.
I think this could happen if the masks don't have the same size as the images.

1. check that the images and the masks are the same size
2. check that the images and the masks are correctly paired in the sequence.xml file, in case not all your cameras have the same size.

You can have a look here to understand how the masks are processed.

First, all the pixels outside of the mask are replaced with the chosen background color. For DTU I used blue, you can use anything else as long as it's different from the color of the object. Black is also fine.
Then, a blur is applied on the mask before copying it to the alpha channel of the image.

During training, the (constant) background color is used as a replacement for an actual background image.
The mask information is only used by the regularizations, with several hyperparameters controlling this.

• OutsideMaskLossWeight: Apply a specific weight on the photometric gradients outside of the mask. Default is 1.0, which means same weighting inside and outside the mask.
• MaskReg: Forces the transmittance to follow what the mask says. It's not very effective and it creates artefacts so I don't use it.
• MaskCarvingReg: The sdf should be > 0 for all the samples along rays originating from outside of the mask. This is very effecting for carving, hence the name.

You can have a look here and there for the implementation details.

Since only a small part of the image is covered by the mask, the ellipsoid initialization is likely to take a long time to converge.

If possible, you should consider using backgrounds images since the quality of the masks is poor.
I think you can generate background images from the video sequences as long as the background is visible in at least one image for every pixel.

[MarilynKeller]

Thanks, indeed the masks had the wrong size.

I created cleaner masks for testing and could adjust the scene with the Gui as you said.

I can now run the reconstruction but for now each camera creates a mesh in a different plane instead of creating one blob. I suspect maybe the calibration data have a problem. I haven't undistorted the images yet, but I don't think it can explain such a discrepancy in the projectt

But do you see any other reason that could lead to that ?

The optimization recording: https://www.dropbox.com/scl/fi/b9r0d2cp8wnv4ch40w3f8/Screencast-from-2025-02-07-18-04-53.webm?rlkey=hdfi9aueec6yytr8kkpt4vouc&st=xfa5pv89&dl=0

The result in volumetric rendering:
https://github.com/user-attachments/assets/fb811f2b-5927-4aba-91ef-7069056b85e7

[btoussai]

Hi,
I think the ellipsoid initialization is too far away from the shape, and the low number of cameras isn't helping.
It should work much better if you initialize with the visual hull.
You can use it if you apply the mask on the images yourself.

1. Create background images. In that case, it would just be blue images with same size as the input images. The background images must be named "cam-{ID}.png" with ID the ID of the camera specified in the xml calibration file.
2. Mask the input images yourself and replace the outside pixels with the background color. This will be the input to the program.
3. Adapt the sequence.xml and config file accordingly: no masks and visual hull initialization.
4. Try to shrink the bounding box around the actual location of the monkey. You can also check "Remove voxels outside field of view" in the left panel, before loading the images. This is useful if you know the shape is fully visible from all the viewpoints.

If you don't want to use the visual hull, you should still shrink the bounding box as the ellipsoid will be a better initialization once it's centered on the monkey.

The distortion is quite significant here. Straight lines in the real world should project to straight lines in the images once distortion is removed.
The k1 and k2 parameters of the calibration file are the coefficient of the distortion polynomial.
I'm using a simplified version of the opencv projection model (only k1 and k2 are handled, the rest must be zero). If your calibration follows the same convention, you can use it as it is. It is very important for the calibration to be accurate for good results, which also includes the distortion. There is an opencv function to undistort images.

You can switch back and forth between the real image and the volume rendering (press 'V') in 2D mode, as well as enable / disable distortion to check the effect of the distortion coefficients.

Since the monkey is projecting to a small area on the images, you should use less levels of details than on DTU: 3 LODs should be plenty.

[MarilynKeller]

After discussion with the data collectors, I indeed had to use another model for distortion as it was a fish eye lens. I am getting closer, now the segmentation, visual hull are working:

Now I wonder where the pink color comes from and why the color are so much poorer than the visual hull, any insight? The longer I run the optimization, the worse the color gets.

This is what the Cinf layer show, I see it is linked to color correction but not sure what it is:

[btoussai]

Hi,

The tail is not in the visual hull despite being clearly visible in the images.
I think this could be a hint that the calibration is still not 100% correct, or that the thresholding values should be adjusted.
You can test this in the 2D mode by looking at the extracted border like here.

The pink color you see is a consequence of the green background: magenta + green = gray.
This should tend to disappear in the later levels of detail as the surface becomes more opaque.
You can try to disable the color correction (and the per-camera latent codes) to make sure this effect cannot be compensated by these.
I think the resolution of the grid is way too low compared to the resolution of the images: 1 voxel ~= 1 pixel

You can send me your data if you would like me to have a closer look.