[RFC] Standard procedures for orchestrating and extending GeoNode docker images

[lucernae]

Hi everyone,

For TL;DR version, I would like you to read the draft of the procedures in:
https://github.com/kartoza/docker-geonode/blob/main/build/docs/DEVELOPMENT.md

Then provide feedback in this thread.

Long explanation about the reasoning

We have used/reused/customized/deployed many GeoNode instance in the past.
I realized that normally we extend GeoNode into our own specific projects.
But this introduces problem over time as our unique instance of GeoNode grows:

• Each deployment don't have common standard. Making it difficult to perform generic deployment task
• Each docker image don't have common standard. Sometimes it's not even production ready (the source code is not included inside the image), and only intended for development
• The docker image is not deterministic. Doing the same build a couple of months later results in a broken image.
• We can't construct a base Kartoza standard images. Each project have it's own improvements, but it doesn't carry over to other project easily
• It's difficult to ask quick deployment or upgrades because people refer to features. For example, project A wants features from projects B, but you can't easily swap docker image from B to A because python migration files is different or some other similar issues. Sometimes project B is even a legacy project where the docker image is not production ready.

I'm drafting a base standard procedures in https://github.com/kartoza/docker-geonode/blob/main/build/docs/DEVELOPMENT.md that tries to address and solve the following problems:

• Docker image can be built and pushed from a central orchestration repo. Even if the source code is on a different project repo. In my opinion, it's easier to include the source code as submodule in this repo and track the development, compared with developing Github Action for each and every project we have (it involves significant time in setting up the credentials and testing/maintaining the action)
• We have a way to override the Dockerfile recipe from the source repo. DevOps can easily help making a production ready recipe, decoupled from the developers workflow. (Some projects don't have project repo because it's one off deployment).
• We have a way to store and make the Dockerfile recipe deterministic.
• We can track which image is based on what geonode version and where to get them.
• We can have a proper git tag that refer to a deterministic recipe
• We can have a proper docker image tag with agreed convention (and have a deterministic canonical tag suitable for deployment)
• Extending a proven build recipe to other project should be easier by making a branch, instead of new repo. The branch will have access to other ops improvement we have done in other project because it will be stored in this repo
• Improving and maintaining build procedure (like bash scripts or Github Action) in one branch/project can offer benefit to other branch/project as well because we can just cherry-pick the changes.

Our repo structure for orchestration (this repo) is slightly different from our usual repo structure for development workflow.
The notable difference (and why I recommend this) are:

• We use trunk based development model instead of usual standard git flow. This is because we want to make the build happen in the cloud (using Github Action) + CI/CD if possible. Git tags will be a deterministic build for ops, while git branches is intended to continuously evolve as devops put commits directly in the branch, or make a new branch for testing the build.
• We use source code structure that I called Depth-Overlay. We organize main source code in a flat directory structure and assign them depth. We treat them as overlays. A file/dir in a higher depth value will override the same file/dir in a source overlay with lower depth value. We use overlay because for ops, it is very common to overlay config file/recipes on top of the same config file/recipes from other sources.
• Project includes other projects as overlays. If Benin project is using our base GEP Geonode and base GEP Geonode is using our base GeoNode image, then we will have 4 flat source directory with depth in increasing order: main , develop, gep, benin. This way, the benin source overlay only contains some files that will override any files with the same relative path from the other sources.
• Pulling improvements from other branch will be easy because normally each branch have it's own source directory. So there will be minimal conflicts. Even if there was, you would normally choose your own changes.
• Main branch/projects can receive improvements from their sub branch/projects by just copying the needed files to their own source overlay. It will not affect the sub branch directly because it's on a different folder.
• We can use the same orchestration structure for our other docker images recipes as well. The benefit, for example to our famous kartoza/postgis, we can have a more deterministic build for different postgres/postgis combination. The naming convention for the docker image and git tag can support deterministic versioning of postgres version + postgis version + the orchestration version.

From the above reasoning, the main idea of this code organization is so that we can generalize our deployment and ops strategy into 3 main phases:

1. Depth-Overlay

We define the source overlay and how they relate. Then we create the merged sources in a separate directory that we
called the merged overlay.

2. Generate

From the merged overlay, we generate any secondary files/structures from the resulting merged configuration. This may include:

• Generating multi stage Dockerfile recipes from a source that don't use them
• Generating documentation from a generic template shared with other branches, but with context specific for the projects
• Generating deployment manifests for specific project/clients that is ready to use immediately. For example, some docker image might have to be pushed into a different repository

3. Evaluate/Execute

In this phase, we actually do the final phase of what the repo is intended to do. For docker-geonode this means building a docker image and push it to repository. For ops related repo, that means deploying the manifests to kubernetes. It can be building a product or doing certain tasks.

From the main phases name above, I recommend calling this approach the DOGE approach.
Once the draft is finalized, we can make toolset that can help setting up this project structures and executing those phases.

[dominicschaff]

To make sure I understand this correctly. We will have the base files (The dockerfile, the base configs and such) in the main branch. And when a project needs to also have this we branch off the main branch to the project branch, make any changes and use that branch going forwards for that project.

This could lead to a project which is based on another project, based on my understanding. Which I think could be a good idea.

The questions that come to mind would be:

1. How often will changes from the main branch need to be synced to the project branches?
2. How often will we need to copy those changes to all of the projects?
3. If we have incompatible changes, I assume we will need to compare them individually to determine what we will do. The main concern would be how often this could happen?
4. Is it common for projects to need to override the base setup files? I assume it is common, otherwise this document would not be needed.
5. I assume we will do the same thing for other base projects, and not only the Geonode setup. Would we then replicate this setup into the other projects?
6. Would we perhaps want to create configure scripts to automatically build up the configs based on config files?

[lucernae]

To answer the question:

1. I haven't decided the interval for the project branches to sync with the main branch. But the idea is they can sync as necessary when the project branch thinks it is time to sync to the main branch. With the regular git fork process from a main upstream repo, developer tends to be discouraged when they want to pull from upstream because there will be many conflicts. But with this approach they know that they only need to check for files that they override, without worrying for git conflicts (because it is in a separate directory). So they can easily pull changes from main branch, test as needed and then decide if they want to proceed or revert the commit without affecting the main source code repository.
2. Same reason as above. As necessary as the project manager thinks
3. Yes, we compare it using a difftool (currently I'm using Pycharm's and VSCode compare command). We know which to compare because we only put files that we care in the project source overlay. This will happen when they pull from the parent branch of the project.
4. Yes, it's common for things like packaging the docker image because some project can have it's own customization, like theme or plugins to install. But this is not supposed to be done for all kinds of repo. Working with source code directly is still best if we are using standard git flow
5. Yes I aim to replicate this setup for other docker repo, like postgis, where we have to support multiple postgis version and postgres combination with the same build and testing recipe.
6. Yes, if you check I already put some basic configuration scripts in the /src/main/bootstrap directory. When it's stable, we can get it out and make a specific repo for this as standalone tools.

[dominicschaff]

For the versioning in the related document, how often will we need complicated version names as described? I can see us using main application and submodule as described. But I don't quite understand how many more levels we would need.

How ever I am quite happy with your suggestion, considering it allows for as many levels as we would need,

[lucernae]

The complicated version name is just for canonical tagging. It can be shortened as the docker hub repo for the project+shorthand docker tag. We only use the full version tag for prod deployment.

To describe the complete dependencies, I'm suggesting to convey that in the docker label or metadata file instead in the docker tag.

[dominicschaff]

One thing I am uncertain about. Will this repo contain the actual source code for the projects? I think it shouldn't but just want to make sure.

Or will this more be about, the changes that will be required per project.

For example geonode allows you to add new applications without touching the base source code. Will this repo contain that new source code. Or will we just link to where the source code is located?

[lucernae]

One thing I am uncertain about. Will this repo contain the actual source code for the projects? I think it shouldn't but just want to make sure.

Yes it shouldn't. But this repo will be tested to adopt that approach first.

For example geonode allows you to add new applications without touching the base source code. Will this repo contain that new source code. Or will we just link to where the source code is located?

Just link the source code, and override only needed files. Some, project need a different project structure that can't be done from forking the original upstream source repo. And here's where the overlay structure come into play.

[dominicschaff]

With the multiple depths setting. How will we know the parent branch? It's obvious for depth 0 and 1, but after that?

I would suggest perhaps with the depth setting to also contain an optional parentBranch or similar setting

[lucernae]

I actually want this to be less complicated. It will just be a single chain of depth. So the parent is just the branch in the previous numerical depth.

[dominicschaff]

For the overlays directory. You mention that we will merge the files. However I assume there might be an order that needs to be followed. How will we enforce an ordering?

[lucernae]

the ordering is from the depth.

currently I only implemented "overwrite" actions so if there exists the same files in both depth 0 and depth 1, the final directory will only contains the file from depth 1.

[dominicschaff]

From your comments then I'm quite happy with this RFC. I think it sounds more complicated that it will actually be, in other words you have thought of the really complicated steps even if we don't use everything.

[tlvu]

Sorry for the late reply. This looks like a lot to wrap my head around and I was swamped lately. Below some reply based on my personal experiences, might not be applicable directly but might give you some ideas.

• The docker image is not deterministic. Doing the same build a couple of months later results in a broken image.

This is probably not avoidable unless you pin every possible dependencies (including all recursive and indirect dependencies). However pinning all dependencies is extremely hard to manage because all the indirect dependencies can change their list of dependencies as well.

So instead of investing on deterministic build, you probably should invest in automated testing. If the new docker build still pass the tests, it should be good to go.

• It's difficult to ask quick deployment or upgrades because people refer to features. For example, project A wants features from projects B, but you can't easily swap docker image from B to A because python migration files is different or some other similar issues. Sometimes project B is even a legacy project where the docker image is not production ready.

I think this is what "feature flags" is meant to solve. You have one single code base and depending on the needs of each project, you activate the corresponding list of features.

Having used your kartoza/geoserver image, I think you basically have implemented "feature flags" by bundling all the extensions in the image and each user will only activate the extension they need.

Having one single code base means all changes should be "backward-compatible". Taking kartoza/geoserver as example again, all the environment var used to influence the behavior of the image, those environment var are like a "stable interface" to configure the image and so they should not be renamed/removed and their associated behavior should not be changed.

Will go on and read https://github.com/kartoza/docker-geonode/blob/main/build/docs/DEVELOPMENT.md and provide feedback here.

[lucernae]

Thanks for your feedback. We really appreaciate it.

So instead of investing on deterministic build, you probably should invest in automated testing. If the new docker build still pass the tests, it should be good to go.

I agree with this. Our biggest issue currently is not knowing the best way (yet) to test, if the test involves asserting the HTML/Web output. For the REST API stuff, we can use python as this is the programming language we are most familiar with.

I think this is what "feature flags" is meant to solve. You have one single code base and depending on the needs of each project, you activate the corresponding list of features.

I was thinking that maybe the "feature flags" should be implemented at the build process level, so we can have more lean image. Then we can package most popular community extensions as a docker image tag variant. Or, let the user build their own customized feature flag activation in their own forked Github Action.

[tlvu]

Our biggest issue currently is not knowing the best way (yet) to test, if the test involves asserting the HTML/Web output.

I have used Selenium in the past to test HTML/Web output. It's one of the popular ones and it's open source. But there are other alternative also if that do not suit you.

I was thinking that maybe the "feature flags" should be implemented at the build process level, so we can have more lean image.

I have experienced both so I'll list their pros and cons and you can decide.

So we have this gigantic Jupyter server image https://hub.docker.com/repository/docker/pavics/workflow-tests (3G compressed on DockerHub, 6G uncompressed on disk, 45 mins to build on DockerHub because Conda is slow and heavy but dependency management is amazing).

That Jupyter image has all the plotting/visualization/calculation libraries we need. It's so gigantic because of all the indirect dependencies. We foresee one day we will have conflicts between those indirect dependencies if we keep adding more libraries to the image. We already have a mechanism to break this monolithic image into multiple smaller images/builds but so far we keep it as-is for its simplicity (both for management and for the end-user).

Advantages of one single image:

• No multi image cascading build to manage: example if we have 1 base image and 3 child images, a rebuild of the base image will have to also force rebuild of all the 3 child images.
• One single image to test/certify.
• Jupyter user do not have to switch between images for their different flavors of notebooks. This is probably the biggest advantage. Switching image means shutdown of the Jupyter server which means the end-user can not have long running calculation notebooks and can not quickly try out multiple "flavors" of the image.

Advantages of multiples smaller image:

• Scale better for more libraries. This is by far the biggest win.
• Each smaller images builds faster. Not sure if the sum of all build time of all base and child images is still faster than the big image.
• Smaller images mean faster pull, less local disk consumption, although this advantage vanish if we need to offer all the "flavors" images on the same host.

Basically a trade-off between scalability (more libraries possible, lower build/pull time) and simplicity (no cascading build management/test and better end-user experience). So we choose to prioritize end-user experience for now until it is technically problematic.

I am not sure how many different "image flavors" you need to produce for all the different projects you handle/manage. How much lower build/pull time is critical to you. How often a project will want to try "options/extensions" from another project and you end up merging two "image flavors" or creating another "image flavor". How frequent does your base image change, which will force the cascading build and re-test of all the child images.