[FEATURE]

[mzpqnxow]

Big caveat: I'm on mobile

Runtime templating of YaML file to complement expand/environment; possible addition of include directive

I would like feedback as to whether you think these ideas have a place as enhancements or if it should be a separate derivative work. They start to stray far outside being "only" an enhanced abstraction around DictConfig

The ideas, roughly- you could define variables in one location in a special block in the YaML file and then reference them as variables elsewhere in the file. It would require two passes to fully load the YaML, or at least that's how I've done it

I've also been considering implementing an include directive, so multiple logging configurations could reference a common one (e.g. a separate formatters.yml file)

Implementation

I have several projects that have a YaML loader that behaves roughly like this:

1. Create a dict with some commonly used dynamic values (e.g. an application name, machine hostname or FQDN, OS family, YYYYMMDD value, etc. Example: {"yyyymmdd": datetime.datetime.now().strftime("%Y%m%d"), "fqdn": socket.getfqdn(), "hostname": socket.gethostname, ...}
2. Load YaML file
3. Pull out one specific section of the file (a dict, usually named globals or variables, usually defined at the top of the YaML file)
4. Apply Jinja2 (or manual) templating over the YaML logging config data (now existing as a dict in Python)

The data provided to the templating engine in # 4 is the preprepared mapping I mentioned in # 1 as well as the k/v pairs from the file itself I mentioned in # 3

This way things like the hostname or OS family are exposed to the logging config sections, even when only using the "safe" YaML loader. You can get the hostname, for example, without running code from the YaML file, using the string {{ hostname }} as part of any YaML string value

Use-Case: Templating

Here's a pseudo-code example. I'm on mobile so bear with me :)

variables:
    app: myapp
    revision: 1.2a
    full_app: '{{ app }}-{{ revision }}'
    ...
formatters:
    ....

handlers:
    ....
    somehandler:
        filename: '{{ full_app }}.log'
    ....
...

This is obviously a very contrived example, and in most deployments could be done via environment variables, so you'll have to be imaginative about what other things this could facilitate

Use-Case: Include Files

Another example of what I'm thinking about doing is implementing a standard include feature- where the loggers, formatters, or any other section would support including configuration info from a logging.d/ directory (or wherever)

This would be a much heavier and invasive change than the simple templating. It could also be done in several ways, with or without a templating engine like Jinja2

Here's my specific use-case; I always have the same formatters section in all of my apps, but the other sections may be very app-specific. In practice this means when I change one, I usually want to change all of them. That causes two pain points:

1. Redundant boilerplate in each app's logger YaML
2. I have to remember to sync the YaML in each project when I make enhancements/changes in one, as I want the formatting to be consistent; this is the case especially for structured loggers and loggers sending to a SOC via something like syslog, where a consistent format is important

It would be nice to reference one common file from the logging YaML of several apps. In my pattern, I install the apps to a venv. Data files go to a venv etc directory, the path being dependable upon the project and app name

I use this pattern for package naming- I'm big into name-spacing for several reasons I won't get into ;)

Visually, package (well, egg/repo) name:

<myorg>.<myproj>.<appname>

Within each app, I generally have a data directory that is installed by setuptools into $VIRTUAL_ENV/etc/<myproj>/<appname>/

For data files that span multiple apps within the same project, I put them in $VIRTUAL_ENV/etc/<myproj>/. It's nice and orderly :)

I would like to use something like this with logging-extras:

$VIRTUAL_ENV/
    etc/
        <myorg>/
            <myproj>/
                common-formatters.yml
                common-handlers.yml
                <app1>/
                    loggers.yml
                <app2>/
                    loggers.yml

Notes

• I would only implement this using "safe" YaML loaders and "safe" Jinja2 loaders. I don't want to make data into code
• The first use-case (simple templating of the value portions within the YaML file) is probably easiest to implement with something like Jinja2, but there are other approaches I'm sure
• The second use-case (an include mechanism) could be probably also be done with the assistance of Jinja2 but there may be a simpler way to do it. Maybe there's even a way native to YaML/PyYAML that I'm not aware of. I've not read any of the YaML specs and have used only the simplest interfaces of PyYaML

This isn't really a concrete plan or proposal. I think what I'm interested in knowing is whether you think features like this would be too creeping for your vision of logging-extras to send as a PR

If they are beyond what you'd like, I'm OK with forking and creating a derivative work based on logging-extras, with the necessary credit/license but with a different name to avoid confusion. I don't thing logging-extras-extras would be a good name :)

One way would be to implement it as an extras in setuptools (pip install logging-extras[templating]?) but that would still pollute your code base

If you think this direction is somewhere you'd like to go, I can work on it here with your input as a 'v2' (or just plain old 'dev') branch, whichever you preferred

Either way, thanks again, I've really simplified my logging with the YaML support your project provides, and the environment var and shell expansion have been key as well. It's made a huge difference for me and I appreciate the work you put into it. Also, despite what my ideas may suggest, I really appreciate how simple you kept the implementation- there's certainly tremendous value in keeping it simple and with a well-defined goal/vision

Any feedback you have the time to provide is appreciated. I already have implemented some of this separately but I won't send any PRs unless you're interested

[zobayer1]

Hi, thanks for such well-put-together thoughts.

Templating

The first use case, templating, can be extremely handy. I was working on a Redis logger plugin for this library, and this would go nicely with the key naming patterns where I would need to create Redis keys for holding logs based on date patterns to simulate the "rollover".

Include Files

Feature for including other Yaml files can come in handy. One use-case I needed was to separate some of the keys to a different Yaml file. A logical hierarchy is even possible to implement. I have faced use cases where I was emitting logs to something like a firebase, and they would require you to have a security file for identification and authentication.

Regarding Implementation Changes

I haven't looked or worked much with the Jinja2 framework (even though I professionally work with python, shame :P ), but Yaml is quite limited when it comes to flexibility. Also, the PyYaml library doesn't provide a whole lot of ways to alter the behavior without accessing its private methods and properties, which I don't really like.

All of your ideas are great and quite practical in my opinion. The thing you mentioned about double pass over the Yaml, I think it can be doable with a single pass, but then the order the keys appear in the file will have to be fixed. For example, patterns/placeholders must be defined before they are used. Just like we are doing for the environment variable.

Please feel free to send some PRs, I am very interested to see your implementation, especially with Jinja2. I myself have to look into it. Also, this is all under MIT, please feel free to maintain separate work as well, it's up to you how much you'd like to share :)

Again, thanks a lot.

[mzpqnxow]

Thanks! I'm looking forwarding to sending stuff over as I have time

[mzpqnxow]

I have a few updates on this if you're interested, though nothing to PR or push to a branch on my fork yet. If you have any comments or suggestions I'm happy to hear them

Templating

I recently started experimenting with using a different approach to the Jinja2 templating than I was previously using. I added a constructor for the tag tag:yaml.org,2002:str (which fires on all strings) and have the constructor perform the Jinja2 template rendering dynamically at load-time, consistent with how the environment variable and home directory expansion is done in logging-extras. Previously, I was loading the YaML file, then walking it and applying the template rendering to each string value.

I'm not really sure as to which may be preferred. The only downside to doing it dynamically with a constructor is it requires more code to use "self-templating", it will have to be a class rather than a simple function (which is probably fine as that's what logging-extras does already)

When I say "self-templating" I just mean something like this- accessing variables defined previously in the YaML file. Maybe it's a pattern that shouldn't be encouraged, but it may be useful or even required for some of the common include functionality

vars:
  var1: test
  var2: test2
main:
  var3: "this is {{ var1 }}"

Including

I decided to implement this dynamically as a constructor also, so the syntax is:

main:
  loggers: !include common_loggers.yaml

This works well enough and is probably superior to doing it as a post-load step.

That's all I have to add for now, the code is functional but hasn't been cleaned up or put into logging-extras, I'm only using it in a separate project at the moment

[mzpqnxow]

Also, because you were interested in the Jinja2 bit, here's a contrived (but real) example. This is the version that uses the string tag to template dynamically:

    ...
        cls.add_constructor('tag:yaml.org,2002:str', cls._ctor_jinja2)
    ...

    def _ctor_jinja2(self, node: Any) -> str:
        """Simple Jinja2 template rendering of a node value

        Notes
        -----
        Invoked internally via PyYaML when encountering string values

        Parameters
        ----------
        node:
            PyYaML YaML node object

        Returns
        -------
        str
            The node value after Jinja2 template evaluation has been applied

        Notes
        -----
        The `data` property contains the variables available to the rendering engine
        
        Example of how that works:

        Data dict:
            data = {'var1': ['a', 'b', 'c']}

        Original YaML:
            test:
              var: '{{ var1 }}'

        Results in the equivalent of:
            test:
              - a
              - b
              - c

        """
        if self.data is None:
            logger.debug('No data dict, skipping templating!')
            return node.value
        # TODO: Allow specification of a safe or unsafe Jinja2 Template context
        template = jinja2.Template(
            node.value,
            undefined=jinja2.StrictUndefined if self.strict_undefined else jinja2.Undefined)
        data = {**os.environ, **self.data}  # Make the environment vars accessible directly with simple syntax
        value = template.render(data)
        if value != node.value:
            logger.debug(f'Before: "{node.value}"')
            logger.debug(f'After:  "{value}"')
        return value

[mzpqnxow]

I checked in an example of how I did !include support. I did not checkin the template rendering as it requires more invasive changes to function in the way I implemented it, as an extension of yaml.Loader

I included a test driver and two test YaML configs as well

There's some basic security checks but they're really pretty basic, just checking to ensure that the included file is at or below the specified directory. It normalizes the path before checking so silly stuff like ../../../ shouldn't "bypass" the check, but it should still just be considered best-effort

This is implemented so that the included file is included directly in-place. If you wanted to support something like:

somelist:
  - !include listfile1.yaml
  - !include listfile2.yaml

As opposed to:

somelist: !include listfile.yaml

... it would need some more logic, to know that it should flatten lists in each file into one. The actual flattening is obviously not difficult but there would need to be a directive (or heuristics of some sort) to signal that it should be flattened, which could get a little ugly. Maybe it's not necessary though and it would be a fine limitation to function this way

You can look at the commits in the attached PR, but the sample looks like:

# common-handlers.yaml
---
console:
  class: logging.StreamHandler
  formatter: simple
  stream: ext://sys.stdout
file_handler:
  class: logging.FileHandler
  filename: ${LOGGING_ROOT:.}/${LOG_FILENAME}
  formatter: simple

# logging.yaml
---
formatters:
  simple:
    format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
handlers: !include common-handlers.yaml
loggers:
  test_logger:
    level: DEBUG
    handlers:
      - file_handler
    propagate: no
root:
  level: NOTSET
  handlers:
    - console

[mzpqnxow]

I posted the full version of what I'm using in my project here

It's a bit over-engineered and makes use of meta-classes, but it works (both the templating and the include directive)

It does not have support for expansion of ~ but it does support environment variables; however, the environment variables need to be referenced using Jinja2 syntax, so it would be '{{ HOME }}' instead of $HOME. It wouldn't be too terribly difficult to support both, though

You'll see what I mean about it not being an simple paste into logging-extras "as-is", mainly due to the metaclass usage and extension of YamlLoader

[mzpqnxow]

What are your thoughts on having a directive that behaves similarly to !include, but explicitly loads the values into the template dictionary? For example:

vars:
  - !variables some-template-vars.yml
main:
  a: '{{ variable_defined_in_template_vars_yml }}'

Another way to do this is to have vars (or some section specified by the caller) treated as special and loaded first. When loaded, the values could all go into the template rendering data dictionary. They can then be available while loading the rest of the file. Of course ordering is important, as you mentioned, but that could be a documented caveat- or the values could be applied using the two-pass method, by iterating over the resulting object after it is loaded

Keeping the syntax simple is probably a better idea, I think, so maybe a special directive is a bad idea