Add rclone as encryption proxy for cnpg backups

[mikeshootzz]

Summary

This deploys rclone alongside vshnpostgresqlcnpg to act as an intermediate encryption proxy

Checklist

• Update tests.
• Link this PR to related issues.
• Merge with /merge comment.

Component PR: vshn/component-appcat#1058

[github-actions]

• Check out Your new Comp-Functions diff at: https://git.vshn.net/appcat/comp-func-pipelines/-/pipelines/651546

[github-actions]

• Check out Your new Comp-Functions diff at: https://git.vshn.net/appcat/comp-func-pipelines/-/pipelines/651549

[github-actions]

• Check out Your new Comp-Functions diff at: https://git.vshn.net/appcat/comp-func-pipelines/-/pipelines/652467

[github-actions]

• Check out Your new Comp-Functions diff at: https://git.vshn.net/appcat/comp-func-pipelines/-/pipelines/653762

[github-actions]

• Check out Your new Comp-Functions diff at: https://git.vshn.net/appcat/comp-func-pipelines/-/pipelines/653766

[github-actions]

• Check out Your new Comp-Functions diff at: https://git.vshn.net/appcat/comp-func-pipelines/-/pipelines/653883

[github-actions]

• Check out Your new Comp-Functions diff at: https://git.vshn.net/appcat/comp-func-pipelines/-/pipelines/654459

[github-actions]

• Check out Your new Comp-Functions diff at: https://git.vshn.net/appcat/comp-func-pipelines/-/pipelines/654528

[github-actions]

• Check out Your new Comp-Functions diff at: https://git.vshn.net/appcat/comp-func-pipelines/-/pipelines/654961

[TheBigLee]

LGTM.
Nitpick: Would have been nice to have the backup encryption layer optional

[mikeshootzz]

LGTM. Nitpick: Would have been nice to have the backup encryption layer optional

@TheBigLee I've thought about that. Since K8up-based backups are always encrypted it made the most sense to keep that behavior.

[mdnix]

Did you test how rclone holds up when there's actually a lot of data to backup? No limits are being set here so I'm assuming the charts default values apply here which are memory: "512Mi" and cpu: "25m".

[mikeshootzz]

Did you test how rclone holds up when there's actually a lot of data to backup? No limits are being set here so I'm assuming the charts default values apply here which are memory: "512Mi" and cpu: "25m".

@mdnix Yes. I tested it with about 5GB of data and that worked without issues. Of course, I wasn't able to test hundreds of GB. I'm still a bit concerned about the multipart upload limitations with rclone, but this should be enough for the foreseeable future.

[mdnix]

Did you test how rclone holds up when there's actually a lot of data to backup? No limits are being set here so I'm assuming the charts default values apply here which are memory: "512Mi" and cpu: "25m".

@mdnix Yes. I tested it with about 5GB of data and that worked without issues. Of course, I wasn't able to test hundreds of GB. I'm still a bit concerned about the multipart upload limitations with rclone, but this should be enough for the foreseeable future.

Could we please test this with usage that more closely reflects a customer's production workload? 5 GiB is not enough to test this imo. We need to know how reliable it is now, not then when in prod backups suddenly stop working. :)

[mikeshootzz]

Did you test how rclone holds up when there's actually a lot of data to backup? No limits are being set here so I'm assuming the charts default values apply here which are memory: "512Mi" and cpu: "25m".

@mdnix Yes. I tested it with about 5GB of data and that worked without issues. Of course, I wasn't able to test hundreds of GB. I'm still a bit concerned about the multipart upload limitations with rclone, but this should be enough for the foreseeable future.

Could we please test this with usage that more closely reflects a customer's production workload? 5 GiB is not enough to test this imo. We need to know how reliable it is now, not then when in prod backups suddenly stop working. :)

I'll do some more testing on the lab, sorry.

[mikeshootzz]

This did not scale as expected and did not prove as viable for production use cases.