Very Difficult to find Information + Broken Links in Documentation

[luckylinux]

Hi 😃. I was looking into running a PyPi-like Server for hosting an Internal / Private Python Package Registry at work. This Project appeared to be very promising in Terms of Features compared to e.g. Devpi or other Projects like that.

However, whether it's due to my lack of Knowledge in Kubernetes (I only plan on deploying this as a docker-compose.yml File, possibly we'll see about HA or Kubernetes later) or just some Links that broke after an update in the Codebase ... Basically I find it very difficult (to some extent even Impossible) to find any Information.

I think I got to a reasonable Level in terms of Docker/Podman and Compose Files so far, yet
I'm currently stuck at starting Docker Compose and the PostgreSQL Database.

I'm running in Windows 11 with Docker Desktop (we have a License for that at Work) using Bind Mounts. The first obstacle was to touch some Files (IIRC passwd for PostgreSQL and settings.py for the pulp Server Service), otherwise Docker would complain that File is mounted as Folder or vice versa.

Once I managed to fix that, PostgreSQL Container kept complaining that it couldn't find the postgres User. The only Way to solve that and get PostgreSQL to initialize its Database was to REMOVE the passwd File Volume Mount. I actually NEVER used the passwd File with a PostgreSQL Container before so I'm not really sure what it's meant to do. I usually use Podman Secrets (or on some Projects don't support them, the corresponding Environment Variables), but I couldn't find anything in the Documentation that clearly states how to configure the Database using passwd File (as outlined in the Quickstart Guide) or without passwd. Is that meant to contain a single Line for the postgres User in Linux /etc/passwd/ File Format ? Or which Accounts does it require to be Defined there ?

It could be one Issue with WSL2 and Docker Desktop, so I'll also try directly on an Ubuntu 24.04 Virtual Machine on HyperV to rule that out. But the main Mystery based on what I could (not) find is how does pulp-minimal Services determine the Host/Port/Username/Password to connect to PostgreSQL ?

I would expect at least one of the following Options for configuring pulp-minimal within docker-compose.yml:

• Docker Secrets (POSTGRES_USER_FILE, POSTGRES_PASSWORD_FILE, ...)
• Environment Variables like for postgresql Container itself (POSTGRES_USER, POSTGRES_PASSWORD, ...)
• settings.py but unsure how exactly it's supposed to be configured, what Options are available (might be "standard" Django Options, but I am not familiar with django, so it's really a Guess Game)

The Reference Guide doesn't seem to contain any PostgreSQL Setting (it contains Redis Settings though).
The following Links are broken:

• https://pulpproject.org/pulp-operator/#configuring-pulp-operator-to-use-an-external-postgresql-installation
• https://pulpproject.org/pulp-operator/#configuring-pulp-operator-to-use-an-external-redis-installation
• https://docs.pulpproject.org/pulp_operator/configuring/storage/
• https://pulpproject.org/pulp-oci-images/#configure-pulp-to-talk-to-the-database-fedora-36-and-ubuntu-2204-only
• https://docs.pulpproject.org/pulp_operator/pulp/#database

Looking at the PULP Database Configuration but it's only mentioning Kubernetes.

The only thing that remotely resembles a Syntax that I can recognize would be the Ansible Role for Pulp Database (https://docs.pulpproject.org/pulp_installer/roles/pulp_database/). But it seems more related to Ansible than anything else 😞.

There's probably an Assumption within the Project that everybody is using/going to use Kubernetes ? Not sure ... I find the Kubernetes learning Curve quite steep 😞.

Thanks in advance for your Help ☺️. I hope it's not a silly Question 😞

[gerrod3]

@luckylinux Have you read the docker/podman-compose example? https://pulpproject.org/pulp-oci-images/docs/admin/tutorials/quickstart/#podman-or-docker-compose

It contains an example compose.yml and the necessary files needed to start up Pulp with compose. I think most of our users who use compose are using this example file.

[luckylinux]

@gerrod3: Thanks for your Reply.

I was following that Guide actually but I've never needed to use all of the "extra" Files in any of my previous Deployements for other Containers, so I actually just downloaded & tweaked the compose.folders.yml File.

Turns out indeed there is a settings.py and postgres/passwd File in the Repository at that Location.

The DB Connection data is indeed illustrated in settings.py under the DATABASES Variable.

I'm still a bit puzzled about postgres/passwd because, as stated before, I never ever ever needed to configure a passwd File for a Container. Is this due to it being an old File or what could possibly be the Reason ?

By the Way, is the project still alive or is it getting deprecated ? I couldn't see much activity in the Issues Section and the Broken Documentation seemed to also suggest that 😕

[gerrod3]

I'm pretty sure the reason for the passwd file is to get the postgres image to run as uid 26 (0006127), this way the data for the database can be compatible with the multiprocess image (pulp/pulp). I'm pretty sure the intention is that if users want to start with our all-in-one image and then want to switch to the minimal image for more customization they can do so with the same data, that's what the compose.folders.yml file is for.

Of course the compose.folders.yml file is a bit out of date as we haven't maintained it, so I wouldn't recommend using it right now.

Yes this project is very much alive, this repository is just how we deploy Pulp. The majority of our development goes on here: https://github.com/pulp/pulpcore. I'll take a look at the broken links, we transitioned our docs into one site last year and there are still some left overs hanging around.

[luckylinux]

Alright, thanks 👍 . I'll give it a try tomorrow.

I also cross-checked briefly with compose.yml (using Docker Volumes) but I really do NOT like that (Docker Volumes), still they didn't seem that much different. They are a Nightmare to Backup, access to Edit, etc. Yes, Bind Mounts have their own set of Issues, but I never managed to get confortable with Volumes. Maybe once you are used to them it isn't so bad ... For one they tend to get deleted when you do e.g. podman system reset so of course you must be aware of that. Bind Mount I have full control (even though postgres UID - as in the normal Image - typically results in the podman user NOT being to access those Files, you will need root).

I'm still puzzled about passwd but I guess it kinda makes sense (even though I'd Imagine you should be able to do achieve that using User/UID Mapping)

EDIT 1: If anything, the PostgreSQL Version looks unusually Old (I think I usually run 16.x).

[gerrod3]

I don't think there are any restrictions on using bind mounts over volumes besides the normal UID issues on who can access what. If you want to experiment and post the results we will gladly take it and add it as an example to the docs. Right now the compose example is set up for minimal effort needed to get running. We could add some more documentation on how to customize your install.

For your case you probably don't need the passwd file so you can remove it from the postgres volume mount. If you want to run your postgres image with a specific UID I would recommend reading their docs: https://github.com/docker-library/docs/blob/master/postgres/README.md#arbitrary---user-notes

Also, for the postgres version you should be able to upgrade to a newer version if you want. We are on version 13 because some of our stakeholders use old versions, so we are slow to upgrade.

[luckylinux]

So after a lot of Frustration due to Bind Mount Permissions (especially loading the Symmetric Key File, finally "fixed" by setting user: root then changed that to user: <mymainuserid> in compose.yml File), I got a proof of concept working and could get the API Status Page showing up on my local VM 😃.

I'm still a bit puzzled that docker does NOT have anything like podman unshare to basically "Fix" these Permissions Issues as an unprivileged User, since I don't have root or sudo access to that Server 😞.

Surely using Docker / Podman Volumes is the recommended Way, but I still cannot figure out how to do simple stuff like editing a File inside a volume (when the Container refuses to boot of course 🤣).

Of course you can stop everything, mount the Volume into another Container (e.g. busybox, I think this is what also restic Backup Container do, otherwise you'd mount the Volume into another Container and tar it IIRC), etc, but it seems so complex overall compared to a relatively simple local Folder 😕.

[luckylinux]

For anybody with the same Issue getting "locked" due to Docker setting root ownership on the Folders, you can fix that (just had to do that Today 🤣) by using e.g. the busybox Docker Container Image (which runs as root by Default).

Then it's only a Matter of running that and mounting the required Bind-Mount Directories with -v /host/path:/mnt/path:z and do a chown -R <userid>:<userid> /mnt/path.

It's also possible to do more selective stuff using find /host/path -user root -o -group root and then do a chown based on that.