BLD: GitFS configuration

[westurner]

[westurner]

see: https://github.com/saltstack-formulas/salt-formula

[westurner]

salt-formula

| Src: https://github.com/saltstack-formulas/salt-formula

The salt::master::gitfs_remotes pillar value contains a list of git repositories (and configuration): https://github.com/saltstack-formulas/salt-formula/blob/master/pillar.example#L24

[westurner]

| Docs: https://docs.saltstack.com/en/develop/ref/configuration/master.html#gitfs-remotes
| Docs: https://docs.saltstack.com/en/latest/topics/tutorials/gitfs.html
| Docs: https://docs.saltstack.com/en/latest/topics/tutorials/gitfs.html#per-remote-configuration-parameters
| Docs: https://docs.saltstack.com/en/latest/topics/tutorials/gitfs.html#branches-environments-and-top-files

[westurner]

The salt::master::gitfs_remotes pillar value contains a list of git repositories (and configuration): https://github.com/saltstack-formulas/salt-formula/blob/master/pillar.example#L24

Is this more or less flexible than the alternatives:

• clone_everything.sh
• clone_everything.sh --recursive (git submodules)
• pyrpo.py -s . -r sh | tee pyrpo-iso8601.sh.log #needstests
• Salt GitFS
  • What is the prefix for where these repos are cloned in a (master|masterless|minion|salt-ssh) setup?

[westurner]

Is this more or less flexible than the alternatives:

GitFS is more flexible (e.g. with branch mapping)

• Is branch name mapping on by default?

[westurner]

Is branch name mapping on by default?

| Docs: https://docs.saltstack.com/en/latest/topics/tutorials/gitfs.html#branches-environments-and-top-files

BRANCHES, ENVIRONMENTS, AND TOP FILES

When using the gitfs backend, branches, and tags will be mapped to environments using the branch/tag name as an identifier.

There is one exception to this rule: the master branch is implicitly mapped to the base environment.

So, for a typical base, qa, dev setup, the following branches could be used:

master
qa
dev

top.sls files from different branches will be merged into one at runtime. Since this can lead to overly complex configurations, the recommended setup is to have a separate repository, containing only the top.sls file with just one single master branch.

To map a branch other than master as the base environment, use the gitfs_base parameter.

gitfs_base: salt-base

The base can also be configured on a per-remote basis.

[westurner]

• Where are the logs for the GitFS repository operations? (e.g. clone, pull, incoming, checkout)

[westurner]

fileserver_backend does fall-through lookup.

so, e.g. with the following configuration, a locally cloned module (formula) would preempt the GitFS configuration
(resulting in (n*env/branchmap)+(1*local clone)).

fileserver_backend:
  - roots
  - git

[westurner]

• For testing, how do I ensure that GitFS uses the same versions of upstream packages (for test builds over there as for test builds on my machine)?
  • You would need to pin versions in the salt master GitFS config
    • and restart / HUP / reload?.
    • branchname ~= revname = (gitfs_remotes::[uri]::base (per-remote), gitfs_base (global; <base->master|qa|dev|prod>)
  • Alternatives:
    • Pin versions with git submodules
    • something like https://github.com/westurner/pyrpo

[westurner]

What do I think I need here?

• An external versions.txt (as with buildout)
• Different ACLs for each formula git fork
  • How do I ensure timely review of (suggested) incoming formula changes?
    • How do I check for / track upstream changes in packages and sources?
      • Formula dependency management saltstack/salt#12179 salt formula metadata and dependencies
      • ENH: GPG signatures, branch-environment map (GitFS/HgFS workflow) saltstack/salt#12183 [GitFS] code signing

[westurner]

What do I think I need here?

So, my tendency is to punt and say "if this was a shell script I could just [...] in like [...] commands" [with quoting and interpolation];
or a more linear-scripty configuration management tool like chef or ansible;
but, the real utility of working with [salt formulas, puppet modules], IMHO,
is that you can fairly easily port the same formula to different platforms; reusably.

[westurner]

So, my tendency is to punt and say "if this was a shell script I could just [...] in like [...] commands" [with quoting and interpolation];

Advantages of [salt] over shell scripts:

• logged auditability every n minutes
  • check func outputs
• consistent variables to flex for [salt grains]
• strongly encouraged (SHOULD) separation of data and code (YAML, Jinja)
  • less risk of quoting and string transformation errors
• defined semantics for porting from [salt [yaml+jinja]] to Python functions
• async transport bus [salt zmq, raet, ssh]

[westurner]

Salt Pillar backends: https://docs.saltstack.com/en/latest/ref/pillar/all/

cmd_json    
cmd_yaml    
cmd_yamlex  
cobbler 
django_orm  
ec2_pillar  
etcd_pillar 
file_tree   
foreman 
git_pillar  
hg_pillar   
hiera   
libvirt 
mongo   
mysql   
pepa    
pillar_ldap 
puppet  
reclass_adapter 
redismod    
s3  
svn_pillar  
varstack_pillar 
virtkey

| Src: https://github.com/saltstack/salt/tree/develop/salt/pillar
| Docs: https://docs.saltstack.com/en/latest/topics/tutorials/pillar.html
| Docs: https://docs.saltstack.com/en/latest/topics/pillar/#viewing-minion-pillar
| Docs: https://docs.saltstack.com/en/latest/topics/tutorials/pillar.html#setting-pillar-data-on-the-command-line

[arnisoph]

The Salt Package Manager might be a solution for this issue in the future. I think it's still in a very early state. I'll file some feature requests.. :)

#24896 (PR)
#25210
#25211

https://docs.saltstack.com/en/develop/topics/spm/