postgres-backup-s3 not updated + CD pipeline

[scriptcoded]

Hey, nice repo!

It seems that postgres-backup-s3 is not up to date on Docker Hub. The last commit was just two months ago, but Docker Hub says "Last pushed 4 years ago". Any chance it's been missed?

Also, I saw in #57 (comment) that you were looking into setting up a CD pipeline pipeline to automatically release new versions of your images. Is this something that you've gotten around to do, or are you open to solutions? I'd be happy to take a look at it, but don't want to do double work in case it's already done 😄

Cheers!
Malcolm

[iloveitaly]

I haven't had the time to update docker hub :(

[scriptcoded]

@iloveitaly Are you maintaining the images on Docker Hub? As mentioned I'd be willing to set up a CD pipeline.

[iloveitaly]

@scriptcoded if you could set up a CD pipeline, that would be amazing. Yup I'm helping with maintenance here!

[scriptcoded]

@iloveitaly I'll mock something up and raise a PR. Going on vacation at the end of the week, so we'll see when I manage to get it done! :)

[scriptcoded]

@iloveitaly

I’ve thought a little bit about how this would best be done and have come up with three alternatives, each building upon the previous.

Just for clarity, when I say “package” I’m talking about one of the directories residing in the root of this repository.

Alternative 1

It seems that currently all Docker Images in this repo is pushed to the :latest tag on Docker Hub. This would be simple to recreate in a CD pipeline as we could just check which package have been modified in a certain commit and push a new version of :latest for that package. While that’s very easy to implement the big downside is that usage of :latest is generally discouraged and makes it impossible to pin dependencies in a production environment.

Alternative 2

So we have to solve the issue of maintaining multiple versions of packages in a monorepo. For tracking which version each package is on I can see three viable options:

1. Each package has a version.txt file (the name doesn’t matter of course) whose sole purpose is to track the version of the package.
2. There is a versions.json file in the root of the project containing the versions of all packages.
3. We track versions of each package using Git tags formatted like postgres-backup-s3/v1.2.3. An issue with this approach is that the latest version displayed on GitHub will always be the latest version of the last updated package, as there is no way to display versions for multiple tags.

Personally I prefer option 3 as it’s Git-native and makes it easier for the developer to push a new release as the CD flow could be triggered when a new tag is created and pushed: git tag -a postgres-backup-s3/v1.2.3 -m "postgres-backup-s3 version 1.2.3". For options 1 and 2 we’d probably have to check whenever the version files are modified and check if the version has changed.

Regardless of the approach we would then push the following tags to Docker Hub:

• postgres-backup-s3:v1.2.3
• postgres-backup-s3:v1.2
• postgres-backup-s3:v1
• postgres-backup-s3:latest

By pushing different semver “levels” we allow users to receive only patch or minor updates while staying on the same major/mino version.

The added amount of tags would not increase the amount of images being built or the amount of storage being used on Docker Hub (even though I’m pretty sure they don’t limit storage) as each tag is just a reference to the same image, just like Git tags.

Alternative 3

Having versioned, up to date images on Docker Hub is already great, but there is still one issue that persists. Some images target specific versions of software by nature. Take postgres-backup-s3 for example. It uses postgresql-client whose version must align with the version of Postgres being targeted, but to use this backup image you must just hope that the currently released version uses the same version of postgresql-client as your database.

So ideally to solve this we would release tags like this for each image: postgres-backup-s3:{postgresVersion}-v{imageVersion}. This would allow the user to specify which version of Postgres to target.

I haven’t played around with this yet, but I’d imagine we’d have some form of manifest file in each package that needs this feature. That file would contain a list of arguments to use when building the Dockerfile. So the manifest might look something like this:

{
  "postgresVersion": [14, 13, 12]
}

And the Dockerfile would then be built three times with that arg being passed:

docker build --build-arg postgresVersion=14 .
docker build --build-arg postgresVersion=13 .
docker build --build-arg postgresVersion=12 .

---

I’d love to hear your thoughts and discuss which direction you’d like to go. As someone using dockerfiles from this repo myself I’d very much like to see alternative 3 being implemented, but I fully understand if it adds too much complexity to the project.

Cheers,
Malcolm

[iloveitaly]

Hello! Thank you for this detailed response. I agree that option #3 is the best, but it would add additional complexity to the project that I'm not sure is warranted given how infrequently things are updated here (hopefully this change will help improve velocity here!). Why don't we start with option #2. I like the third option (git tags) but the issue I see with that approach is tags can only be managed by users who have write access to the repo and you can't include tags in a PR. For this reason I think a package-level versions.txt would be best. Let me know if you see this differently, I might be missing something with what is possible via tags.

…

On Tue, Mar 22, 2022 at 5:35 AM Malcolm Nihlén ***@***.***> wrote: @iloveitaly <https://github.com/iloveitaly> I’ve thought a little bit about how this would best be done and have come up with three alternatives, each building upon the previous. Just for clarity, when I say “package” I’m talking about one of the directories residing in the root of this repository. Alternative 1 It seems that currently all Docker Images in this repo is pushed to the :latest tag on Docker Hub. This would be simple to recreate in a CD pipeline as we could just check which package have been modified in a certain commit and push a new version of :latest for that package. While that’s very easy to implement the big downside is that usage of :latest is generally discouraged and makes it impossible to pin dependencies in a production environment. Alternative 2 So we have to solve the issue of maintaining multiple versions of packages in a monorepo. For tracking which version each package is on I can see three viable options: 1. Each package has a version.txt file (the name doesn’t matter of course) whose sole purpose is to track the version of the package. 2. There is a versions.json file in the root of the project containing the versions of all packages. 3. We track versions of each package using Git tags formatted like postgres-backup-s3/v1.2.3. An issue with this approach is that the latest version displayed on GitHub will always be the latest version of the last updated package, as there is no way to display versions for multiple tags. Personally I prefer option 3 as it’s Git-native and makes it easier for the developer to push a new release as the CD flow could be triggered when a new tag is created and pushed: git tag -a postgres-backup-s3/v1.2.3 -m "postgres-backup-s3 version 1.2.3". For options 1 and 2 we’d probably have to check whenever the version files are modified and check if the version has changed. Regardless of the approach we would then push the following tags to Docker Hub: - postgres-backup-s3:v1.2.3 - postgres-backup-s3:v1.2 - postgres-backup-s3:v1 - postgres-backup-s3:latest By pushing different semver “levels” we allow users to receive only patch or minor updates while staying on the same major/mino version. The added amount of tags would not increase the amount of images being built or the amount of storage being used on Docker Hub (even though I’m pretty sure they don’t limit storage) as each tag is just a reference to the same image, just like Git tags. Alternative 3 Having versioned, up to date images on Docker Hub is already great, but there is still one issue that persists. Some images target specific versions of software by nature. Take postgres-backup-s3 for example. It uses postgresql-client whose version must align with the version of Postgres being targeted, but to use this backup image you must just hope that the currently released version uses the same version of postgresql-client as your database. So ideally to solve this we would release tags like this for each image: postgres-backup-s3:{postgresVersion}-v{imageVersion}. This would allow the user to specify which version of Postgres to target. I haven’t played around with this yet, but I’d imagine we’d have some form of manifest file in each package that needs this feature. That file would contain a list of arguments to use when building the Dockerfile. So the manifest might look something like this: { "postgresVersion": [14, 13, 12] } And the Dockerfile would then be built three times with that arg being passed: docker build --build-arg postgresVersion=14 . docker build --build-arg postgresVersion=13 . docker build --build-arg postgresVersion=12 . ------------------------------ I’d love to hear your thoughts and discuss which direction you’d like to go. As someone using dockerfiles from this repo myself I’d very much like to see alternative 3 being implemented, but I fully understand if it adds too much complexity to the project. Cheers, Malcolm — Reply to this email directly, view it on GitHub <#141 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABE2RY47CXF23NUFL57ZWTVBGWAXANCNFSM5QPYKDKQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[scriptcoded]

@iloveitaly Hey! Just an update here. Unfortunately I've hit a bit of a snag in life, and am currently under a lot of stress. I'll get to this, I promise! It'll just have to wait a while.

Have a nice weekend!

[iloveitaly]

No worries! I hope things get better for you soon.

…

On Fri, Apr 29, 2022 at 5:58 AM Malcolm Nihlén ***@***.***> wrote: @iloveitaly <https://github.com/iloveitaly> Hey! Just an update here. Unfortunately I've hit a bit of a snag in life, and am currently under a lot of stress. I'll get to this, I promise! It'll just have to wait a while. Have a nice weekend! — Reply to this email directly, view it on GitHub <#141 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABE2R57MH6BRJJVLMLKJT3VHPFGXANCNFSM5QPYKDKQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[tholu]

@scriptcoded @iloveitaly Any plans to get the Docker images up-to-date on Docker Hub again?

[scriptcoded]

Hi @tholu! I've since switched jobs and am not actively using this project anymore. If there is still a need for automation from the maintainer, reach out! But I won't be updating things manually 😊