Should confidence be able to deal with 'partial' files?

[akaIDIOT]

For use cases with either lots of configuration, or lots of alternatives for similar things, the concept of partial files or just splitting content over multiple files might make sense. I can think of a couple ways someone might want to do something like this:

• name.00.yaml, name.01.yaml, ..., name.part42.yaml ­— a bit like many shell inclusion configurations and something like apt configuration is done;
• name.yaml/part1.yaml, name.yaml/partx.yaml — bit confusing to have a directory with a .yaml extension, but at least it'll be easy to detect (might be possible without the directory name extension too, might also give a load of false hits maybe?);
• name/concept1/alternative1.yaml, name/concept1/alternative2.yaml, name/concept2/a.yaml, ... — might cause a forest of separate files, might be hard to accurately interpret what the naming of the concepts and alternatives should mean namespace-wise, but this has been spotted in the wild.

All of the above kind of assumes that the goal / modus operandus of confidence is / remains to load a bunch of things into a single set of configuration things. I'm not sure that's what's always used in practice, but I'd say that if the total set of options is 'too much' to keep in memory, we might no longer be talking about "configuration options".

Within this brainstomy space, I think we should hold on to the following things:

• using safe YAML loaders, references to specific types and calling init functions keeps getting security-related issues;
• the entire configuration namespace concept should be 'possible' to load into a different environment, e.g. load the same things using java and end up with functionally the same thing (sort of follows form the first point, but still);
• simple to understand end-result;
• 100% deterministic loading order of anything that's used;

Furthermore, before we'd add / implement something like this, confidence should really get some way to track where every key originates from. #66 makes an attempt at this. Not sure that's good enough as it is, but as the story from files to namespace becomes more complicated, being able to debug the process and the end result becomes more important.

Looking for feedback / ideas / complications / opinions / ... 🤗

[akaIDIOT]

apt and some other linux tools not only employ the first option, but use a variant of the second, like /etc/apt/apt.conf.d/…, where the .d-suffix implies that anything in that directory is sorted lexicographical and then merged. The .d-suffix being widely known makes this a great candidate for something like this.

Interesting to decide what to do when both /etc/name/name.yaml and /etc/name/name.yaml.d/ exist; load both with a clear logical precedence or error out refusing to overlay a .d'd variant over it's basic variant?