Merge imported files into document root

[b0o]

I wanted to be able to import one or more yaml documents into the root of another document, but this doesn't seem easily doable with yglu as-is.

I was able to craft a bit of a hacky way to accomplish my goal - here is what I came up with:

---
# base.yml

colors:
  primary:
    background: "#700"
    foreground: "#ee0"
  normal:
    black: "#2F2A38"
    red: "#FF87A5"
    green: "#6EEB84"

foobar: 123

qux:
  naw: yah
  lolwut:
    - 1
    - 2
    - 3

users:
  - {name: John, age: 32}
  - {name: Jane, age: 23}

---
# colors.yml

colors:
  primary:
    foreground: "#FFF1E0"
  normal:
    black: "#2F2A38"
    yellow: "#FFDF61"

foobar: 456

qux:
  lolwut:
    - 4
    - 5
    - 6

---
# final.yglu.yml

_import: !()
  - !? $import('base.yml')
  - !? $import('colors.yml')

# Merge imports with each other
# primitives, lists: replace, later take precedence
# dicts:             merge, contained primitives and lists handled as above
_imported: !-
  !for ($_._import)({}): !()
    !for $.items(): !()
      !if isDict($[1]):
        !? $[0]: !? ({result => $[1]}).mergeWith({result => $_._imported.get($[0], null)}, $1, $1).get("result", null)
      !if not isDict($[1]):
        !? $[0]: !? $[1]

# Merge aggregated imports with document root
# primitives, lists: values in document root take precedence
# dicts:             merge, values in document root take precedence, contained primitives and lists handled as above
!for $_._imported.items(): !()
  !if $_.get($[0], null) = null:
    !? $[0]: !? $[1]
  !if $_.get($[0], null) != null and isDict($[1]):
    !? $[0]: !? ({result => $_.get($[0], null)}).mergeWith({result => $[1]}, $1, $1).get("result", null)

# nodes declared directly in this file will be merged with any from imported files

foobar: 789

colors:
  primary:
    background: "#000000"

The output is:

foobar: 789
colors:
  primary:
    background: '#000000'
    foreground: '#FFF1E0'
  normal:
    black: '#2F2A38'
    yellow: '#FFDF61'
    red: '#FF87A5'
    green: '#6EEB84'
qux:
  lolwut:
  - 4
  - 5
  - 6
  naw: yah
users:
- name: John
  age: 32
- name: Jane
  age: 23

As you can see, dictionaries are merged in a semi-intelligent way, to allow for overriding nested values without overwriting entire sub-dictionaries. This is what I wanted for my use-case, but might not always be desirable.

I wanted to share my solution here in case it's helpful to anyone else wanting to do the same, and also to ask if something like this could feasibly be integrated into yglu itself, e.g.

!$import('base.yml')
!$import('colors.yml')

foobar: 789

colors:
  primary:
    background: "#000000"

I also wrote a script to inject the yglu code to do the importing whenever a _import: !() line is detected and pass it through yglu: link.

p.s. Thank you for yglu, it is really useful!

[lbovet]

Many thanks for this contribution.
I was pretty sure that such a feature would be the first to be requested :)
As you say, the merge strategy should be adaptable to the cases. We should be able to specify nodes that must be replaced, merged or deleted.
I did some tries in this direction in the "overlay" branch. (https://github.com/lbovet/yglu/blob/overlay/tests/future/overlay.yml).
I am not yet happy with the result, it is too complex. IMHO.
If you have ideas about the syntax users should use to specify the merge strategy, they are welcome.
Also, in the meantime, until there is a native answer implemented in Yglu, it could be interesting to make your solution easily usable for anyone. You could move _imported and _import in a separated reusable file. Users would just have to import this file and call a function.
If you want to try contributing this, feel free to open a PR with a functioning test in https://github.com/lbovet/yglu/tree/master/tests/samples