YAML interpolate

yaml_interpolate is a command-line tool for doing string interpolation inside a YAML file.

That tool exists, since YAML specification not allow string interploation, only alieses and anchors.

That tool is designed to process a single YAML file with both template and data for it. See possible alternatives (1, 2, 3, 4) for other cases / other languages.

Written in C++11 (not in python or ruby, for example), since I find it hard to express logic without strong typing :)

Command-line options

Usage:
  ./yaml_interpolate [OPTION...]

 I/O options:
  -i, --input PATH   Input yaml file path (or 'stdin'). (required)
  -o, --output PATH  Output file name (or 'stdout') (default: stdout)

 Parsing settings options:
      --formats LIST      Formats:
                          moustache:      {{ variable.path }}
                          double_dollar:  $variable.path$
                          double_percent: %variable.path%
                           (default: moustache)
      --separator STRING  Separator between node names (default: .)
      --regexps LIST      Regular expressions to find pattern
                          Notice, that them must contain single capture group

 Help options:
      --help  Print this help

Example of usage

$ yaml_interpolate --input=input.yaml --output=processed.yaml --formats=moustache

strings:
    str1: "Hello"
    str2: "world"

# Interpolated value: "Hello, world!"
greeting: "{{strings.str1}}, {{strings.str2}}!"

---

Advanced usage

Access by index

numbers:
    - 2.71828
    - 6.022e+23
    - value: 3.14

# Interpolated value: "Numbers: [ 2.71828, 6.022e+23, 3.14 ]"
text: "Numbers: [ {{numbers.0}}, {{numbers.1}}, {{numbers.2.value}} ]"

Multiple formats support:

• Inline form (single formats option):

  $ yaml_interpolate --input=input.yaml --output=processed.yaml \
      --formats=moustache,double_dollar,double_percent

• Multi-option form (multiple formats options):

  $ yaml_interpolate --input=input.yaml --output=processed.yaml \
      --formats=moustache \
      --formats=double_dollar \
      --formats=double_percent

strings:
    str1: "Hello"
    str2: "world"

# Interpolated value: "Hello, world!"
greeting1: "{{ strings.str1 }}, {{ strings.str2 }}!" # <-- format: moustache
greeting2: "$strings.str1$, $strings.str2$!"         # <-- format: double_dollar
greeting3: "%strings.str1%, %strings.str2%!"         # <-- format: double_percent

# Interpolated value: "Greetings: Hello, world!, Hello, world!, Hello, world!"
greetings: "Gretings: %greeting1%, $greeting2$, {{greeting3}}" # <-- Composite

Adding custom regular expressions

For example, here is used the next simple regular expressions:

• \%\{(\S+)\} - parse "%{tokens}" pattern
• \$\{(\S+)\} - parse "${tokens}" pattern
• \$\[(\S+)\] - parse "$[tokens]" pattern

Notice, that some characters in regular expressions must be escaped twice!

For example - passing \${\{(\S+)\} option will be interpreted as $\{(\S+)\} (not \$ at start, but $ - not what we want).

It must be rewritten in the next forms:

• \\${\{(\S+)\} - escaping only \$ at start
• \\$\\{(\\S+)\\} - escaping all special characters (safe way, preferred)
• [\$][\{](\S+)[\}] - non-escaping way, wrapping special characters. Interpreted as: [$][\{](\S+)[\}]

Read about escaping characters in shell here.

---

• Inline form (single regexps option):

  $ yaml_interpolate --input=input.yaml --output=processed.yaml \
      --formats=moustache \
      --regexps="\\%\\{(\\S+)\\}","\\$\\{(\\S+)\\}","\\$\\[(\\S+)\\]"

• Multi-option form (multiple regexps options):

  $ yaml_interpolate --input=input.yaml --output=processed.yaml \
      --formats=moustache \
      --regexps="\\%\\{(\\S+)\\}" \
      --regexps="\\$\\{(\\S+)\\}" \
      --regexps="\\$\\[(\\S+)\\]"

constants:
    pi: 3.1415
    e:  2.7182

# Interpolated value: "[ 3.1415, 2.7182, 3.1415, 2.7182 ]"
text: "[ {{constants.pi}}, %{constants.e}, ${constants.pi}, $[constants.e] ]"

Multiple documents processing

Multi-document processing supported - a YAML character stream may contain several documents. Each document is completely independent from the rest.

---
name:  "Tomato"
color: "red"
text: "{{name}} is {{color}}" # <-- Interpolated value: "Tomato is red"
---
name: "Kiwi"
color: "green"
text: "{{name}} is {{color}}" # <-- Interpolated value: "Kiwi is green"
---
name:  "Blueberry"
color: "blue"
text: "{{name}} is {{color}}" # <-- Interpolated value: "Blueberry is blue"

Piping

Piping is possible by redirecting output into stdout - for reading by another tool. For example - redirecting into yq (analogue of jq):

$ yaml_interpolate --input=input.yaml --output=stdout --formats=moustache | yq eval '.some.field' -

---

Reading from stdin is also possible - to use this tool in a pipeline. For example:

1. Passing minified YAML string
2. Processing by yaml_interpolate
3. Extraction from text node: "Hello, world!"

$ echo "{data: {str: world}, text: 'Hello, {{data.str}}!'}" | yaml_interpolate --input=stdin --output=stdout | yq eval '.text' -

Limitations

• You must always specify full node path - relative search not allowed, since the content of a YAML file constitutes a directed graph, not a tree.

  root_node:
      leaf1: "Leaf 1 text"
      leaf2: "Leaf 2 text"
      leaf3: "{{leaf1}} & {{leaf2}}" # <-- Dont works, use '{{root_node.leaf1}} & {{root_node.leaf2}}'
      leaf4: "{{.leaf5.leaf1}} & {{.leaf5.leaf2}}" # <-- Dont works, use '{{root_node.leaf5.leaf1}} & {{root_node.leaf5.leaf2}}'
      leaf5:
          leaf1: "Leaf 6 text"
          leaf2: "Leaf 7 text"

• Be careful and avoid circular depency and recursion.

  root_node: # Dont do this:
      leaf1: "{{root_node.leaf2}}"
      leaf2: "{{root_node.leaf1}}"
      leaf3: "{{root_node.leaf2}} & {{root_node.leaf3}}"

• Nodes keys not transformed, only values.

Dependencies

• cxxopts
  • Commit 5eca8a30012b69b76316b71fa391a89fe09256cb
• yaml-cpp
  • Commit 0d9dbcfe8c0df699aed8ae050dddaca614178fb1

Build

$ mkdir build
$ cd build
$ bash ../build.sh ../