maint: devcontainer

[stuart-cls]

Dockerfile and .devcontainer config to simplify development and possibly deployment. If you open tofu git checkout in a IDE like VS Code/PyCharm, you get a working container development environment without any build difficulties.

I based the Dockerfile on ufo-kit/ufo-core, but it only targets Nvidia systems at the moment (and assumes you have the nvidia-container-toolkit installed on the host). However since UFO uses OpenCL this could likely be adapted to work on other systems.

One improvement I made was to split building the UFO image from the python image, and only copying build artifacts. This reduces the size of the final image and saves re-building UFO (which seems quite stable at the moment). This multi-build file enables building layers with just UFO, UFO+tofu, and a devcontainer: the runtime build should be appropriate for production use.

The requirements-ufo.txt (and exisiting requirements files) could probably be rationalized with a modern pyproject.toml.

See also https://github.com/sarkarchandan/ufo-docker/tree/master

[tfarago]

I would like to bring @MarcusZuber to this. Can you guys post the steps to make this all work which I can just "play" on my machine?

[stuart-cls]

Assuming you have vscode and some kind of docker/podman setup already working (I know that's a large caveat), you can install the Dev Containers extension. Then when you open the tofu repo checked out to this branch in vscode it will prompt you to create the devcontainer automatically. After the container is built, it will relaunch vscode inside the container.

X11/GPU forwarding are pre-configured, GPU requires an nvidia card with the nvidia-container-toolkit installed on the host. You may want to expose some data to test, you can add this to the end of the .devcontainer/devcontainer.json file:

    // User mounts
    "mounts": [
        "source=/scratch,target=/scratch,type=bind"
    ]
}

The https://code.visualstudio.com/docs/devcontainers/containers docs are quite good and include a tutorial and docker setup instructions for various operating systems.

I hope that's helpful!

[sarkarchandan]

Hi, @stuart-cls, thanks a lot of creating the PR.

@tfarago @MarcusZuber I think with some common understanding of the potential use cases we can definitely convert this from draft to merge candidate. It adds to the convenience to developing on top of ufo-tofu without interfering with the core attributes of the software. So, even if you consider merging this added feature your old ways of using ufo-tofu will remain absolutely intact. Just some additional comments.

• When I started looking into it with Docker and NVIDIA CUDA base image I was also thinking about ease of deploying ufo-tofu software stack onto our reconstruction servers and hence ways to integrate dev containers onto our reconstruction pipeline. I had less familiarity with these software back then and I always found installing these from the scratch cumbersome. Although reconstructions went perfectly alright inside container, using docker I ran into the inconvenience of file permissions. I hope that usage of podman already addresses that issue.

• I also wanted to have simpler dev containers in ufo-core and ufo-filters repositories as well and it fits well with the following from Stuarts Dockerfile.

ARG UFO=master
ARG UFO_FILTERS=master
...
RUN git clone --depth 1 https://github.com/ufo-kit/ufo-core.git --branch ${UFO} && \
    git clone --depth 1 https://github.com/ufo-kit/ufo-filters.git --branch ${UFO_FILTERS}

We can mount the directories for these repositories inside the container. If we make some changes in a feature branch in the repository we can build and test them inside the container without tampering with our local workstations. If things look as expected we push the branch and use that branch inside this dev container. Alternatively, I think we an also customize the Dockerfile in a way, that instead of cloning we mount the ufo-core and ufo-filter repositories from host system to this dev container and use it as one central development platform for all the repositories. Version control would be handled from host system.

3. I like the multistage build setup of Stuart, which I did not think of earlier. My Dockerfile was simpler to look at but was bloated with a lot of unnecessary stuff, so image size was big. Only thing that I notice in the current Dockerfile is that there are some repetitions, specifically of environment variables from one stage to another. I think it can be handled using ARG.

Overall, I think it is a good feature to have. It introduces new ways to do things without interfering with the old.