[RFC] Changes needed for integration tests

[lucernae]

We need a way to test the repo changes as proposed here:

I have no familiarity so far with the stack to determine if the images needs to have it's own unit test. So far we have 3 separate image under one repo. It may need a separate discussion.

At the moment, we can focus on an integration tests (tests that makes sure the stack works as one single services).

To do this, we need to have a clear goals on what needs to be tested:

• we should have small/minimum test dataset with predictable result, as the input
• what to test? connections? schema? final results of the data?
• if possible let’s use test framework like python unittest, so it can be described and added easily
• where to test? I guess we can only rely on GitHub action for now, since it uses docker-compose. hopefully it is under the github action test limit.

Requesting for comments from @timlinux @NyakudyaA @Gustry

[lucernae]

I proposed in the slack channel that the dataset for the tests should be stored in a public location that we can control (possibly the pbf files, clip geojson, and the yml mapping files). We can use MinIO datastore instance that Kartoza have for that. Currently not public, but we can arrange the permission later so that it is publicly fetch-able.

[frafra]

I agree. This is totally needed, together with pinning dependencies.