Added utility for loading tiered config from file system. #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

eriq-augustine merged 66 commits into edulinq:main from BatuhanSA:config

Sep 5, 2025

README.md

-Original file line number
+Diff line change
@@ Expand Up @@
     ```
     pip3 install .
     ```
+    ## Configuration System
+    This project provides a configuration system that supplies options (e.g., username, password) to a command-line interface (CLI) tool.
+    The configuration system follows a tiered order, allowing options to be specified and overridden from both files and command-line options.
+    ### Configuration Sources
+    In addition to CLI options, the configuration system loads options from [JSON](https://en.wikipedia.org/wiki/JSON) files located across multiple directories.
+    By default, configuration files are named `edq-config.json`.
+    This value is customizable, but this document will assume the default is used.
+    For example, a configuration file containing the `user` and `pass` options might look like this:
+    ```json
+    {
+        "user": "alice",
+        "pass": "password123"
+    }
+    ```
+    The table below summarizes the configuration sources in the order they are evaluated.
+    Values from earlier sources can be overwritten by values from later sources.
+    | Source   | Description |
+    | :-----   | :---------- |
+    | Global   | Loaded from a file in a user-specific location, which is platform-dependent. |
+    | Local    | Loaded from a file in the current or nearest ancestor directory. |
+    | CLI File | Loaded from one or more explicitly provided configuration files through the CLI. |
+    | CLI      | Loaded from the command line. |
+    The system produces an error if a global or local configuration file is unreadable (but not missing), or if a CLI-specified file is unreadable or missing.
+    #### Global Configuration
+    Global configuration are options that are user specific and stick with the user between projects, these are well suited for options like login credentials.
+    The global configuration file defaults to `<platform-specific user configuration location>/edq-config.json`.
+    The configuration location is chosen according to the [XDG standard](https://en.wikipedia.org/wiki/Freedesktop.org#Base_Directory_Specification) (implemented by [platformdirs](https://github.com/tox-dev/platformdirs)).
+    Below are examples of user-specific configuration file paths for different operating systems:
+     - Linux -- `/home/<user>/.config/edq-config.json`
+     - Mac -- `/Users/<user>/Library/Application Support/edq-config.json`
+     - Windows -- `C:\Users\<user>\AppData\Local\edq-config.json`
+    The default global configuration location can be changed by passing a path to `--config-global` through the command line.
+    Below is an example command for specifying a global configuration path from the CLI:
+    ```sh
+    python3 -m edq.cli.config.list --config-global  ~/.config/custom-config.json
+    ```
+    #### Local Configuration
+    Local configuration are options that are specific to a project or directory, like a project's build directory.
+    Local configuration files are searched in multiple locations, the first file found is used.
+    The local config search order is:
+. `edq-config.json` in the current directory.
+. A legacy file in the current directory (only if a legacy file is preconfigured).
+. `edq-config.json` in any ancestor directory on the path to root (including root itself).
+    #### CLI-Specified Config Files
+    CLI config files are options specified on the command line via a file.
+    These are useful for a common set of options you don’t need every time, such as login credentials for different user.
+    Any files passed via `--config-file` will be loaded in the order they appear on the command line.
+    Options from later files override options from previous files.
+    Below is an example of a CLI specified configuration paths:
+    ```sh
+    python3 -m edq.cli.config.list --config-file ./edq-config.json --config-file ~/.secrets/edq-config.json
+    ```
+    #### CLI Configuration
+    CLI configurations are options specified directly on the command line, these are useful for quick option overrides without editing config files.
+    Configuration options are passed to the command line by the `--config` flag in this format `<key>=<value>`.
+    The provided values overrides the values from configuration files.
+    Configuration options are structured as key value pairs and keys cannot contain the "=" character.
+    Below is an example of specifying a configuration option directly from the CLI:
+    ```sh
+    python3 -m edq.cli.config.list --config user=alice --config pass=password123
+    ```
+    #### CLI Config Options
+    The table below lists common configuration CLI options available for CLI tools using this library.
+    | CLI Option       | Description |
+    | :--------------  | :---------- |
+    |`--config-global` | Override the global config file location. |
+    |`--config-file`   | Load configuration options from a CLI specified file. |
+    | `--config`       | Provide additional options to a CLI command. |
+    | `--help`         | Display standard help text and the default global configuration file path for the current platform. |

edq/core/config.py

-Original file line number
+Diff line change
@@ -0,0 +1,177 @@
+    import argparse
+    import os
+    import typing
+    import platformdirs
+    import edq.util.dirent
+    import edq.util.json
+    CONFIG_SOURCE_GLOBAL: str = "<global config file>"
+    CONFIG_SOURCE_LOCAL: str = "<local config file>"
+    CONFIG_SOURCE_CLI: str = "<cli config file>"
+    CONFIG_SOURCE_CLI_BARE: str = "<cli argument>"
+    CONFIG_PATHS_KEY: str = 'config_paths'
+    DEFAULT_CONFIG_FILENAME: str = "edq-config.json"
+    class ConfigSource:
+        """ A class for storing config source information. """
+        def __init__(self, label: str, path: typing.Union[str, None] = None) -> None:
+            self.label = label
+            """ The label identifying the config (see CONFIG_SOURCE_* constants). """
+            self.path = path
+            """ The path of where the config was soruced from. """
+        def __eq__(self, other: object) -> bool:
+            if (not isinstance(other, ConfigSource)):
+                return False
+            return ((self.label == other.label) and (self.path == other.path))
+        def __str__(self) -> str:
+            return f"({self.label}, {self.path})"
+    def get_tiered_config(
+            config_file_name: str = DEFAULT_CONFIG_FILENAME,
+            legacy_config_file_name: typing.Union[str, None] = None,
+            global_config_path: typing.Union[str, None] = None,
+            skip_keys: typing.Union[list, None] = None,
+            cli_arguments: typing.Union[dict, argparse.Namespace, None] = None,
+            local_config_root_cutoff: typing.Union[str, None] = None,
+        ) -> typing.Tuple[typing.Dict[str, str], typing.Dict[str, ConfigSource]]:
+        """
+        Load all configuration options from files and command-line arguments.
+        Returns a configuration dictionary with the values based on tiering rules and a source dictionary mapping each key to its origin.
+        """
+        if (global_config_path is None):
+            global_config_path = platformdirs.user_config_dir(config_file_name)
+        if (skip_keys is None):
+            skip_keys = [CONFIG_PATHS_KEY]
+        if (cli_arguments is None):
+            cli_arguments = {}
+        config: typing.Dict[str, str] = {}
+        sources: typing.Dict[str, ConfigSource] = {}
+        # Ensure CLI arguments are always a dict, even if provided as argparse.Namespace.
+        if (isinstance(cli_arguments, argparse.Namespace)):
+            cli_arguments = vars(cli_arguments)
+        # Check the global user config file.
+        if (os.path.isfile(global_config_path)):
+            _load_config_file(global_config_path, config, sources, CONFIG_SOURCE_GLOBAL)
+        # Check the local user config file.
+        local_config_path = _get_local_config_path(
+            config_file_name = config_file_name,
+            legacy_config_file_name = legacy_config_file_name,
+            local_config_root_cutoff = local_config_root_cutoff,
+        )
+        if (local_config_path is not None):
+            _load_config_file(local_config_path, config, sources, CONFIG_SOURCE_LOCAL)
+        # Check the config file specified on the command-line.
+        config_paths = cli_arguments.get(CONFIG_PATHS_KEY, [])
+        if (config_paths is not None):
+            for path in config_paths:
+                _load_config_file(path, config, sources, CONFIG_SOURCE_CLI)
+        # Finally, any command-line options.
+        for (key, value) in cli_arguments.items():
+            if (key in skip_keys):
+                continue
+            if ((value is None) or (value == '')):
+                continue
+            config[key] = value
+            sources[key] = ConfigSource(label = CONFIG_SOURCE_CLI_BARE)
+        return config, sources
+    def _load_config_file(
+            config_path: str,
+            config: typing.Dict[str, str],
+            sources: typing.Dict[str, ConfigSource],
+            source_label: str,
+        ) -> None:
+        """ Loads config variables and the source from the given config JSON file. """
+        config_path = os.path.abspath(config_path)
+        for (key, value) in edq.util.json.load_path(config_path).items():
+            config[key] = value
+            sources[key] = ConfigSource(label = source_label, path = config_path)
+    def _get_local_config_path(
+            config_file_name: str,
+            legacy_config_file_name: typing.Union[str, None] = None,
+            local_config_root_cutoff: typing.Union[str, None] = None,
+        ) -> typing.Union[str, None]:
+        """
+        Search for a config file in hierarchical order.
+        Begins with the provided config file name,
+        optionally checks the legacy config file name if specified,
+        then continues up the directory tree looking for the provided config file name.
+        Returns the path to the first config file found.
+        If no config file is found, returns None.
+        The cutoff parameter limits the search depth, preventing detection of config file in higher-level directories during testing.
+        """
+        # Provided config file is in current directory.
+        if (os.path.isfile(config_file_name)):
+            return os.path.abspath(config_file_name)
+        # Provided legacy config file is in current directory.
+        if (legacy_config_file_name is not None):
+            if (os.path.isfile(legacy_config_file_name)):
+                return os.path.abspath(legacy_config_file_name)
+        # Provided config file is found in an ancestor directory up to the root or cutoff limit.
+        parent_dir = os.path.dirname(os.getcwd())
+        return _get_ancestor_config_file_path(
+            parent_dir,
+            config_file_name = config_file_name,
+            local_config_root_cutoff = local_config_root_cutoff,
+        )
+    def _get_ancestor_config_file_path(
+            current_directory: str,
+            config_file_name: str,
+            local_config_root_cutoff: typing.Union[str, None] = None,
+        ) -> typing.Union[str, None]:
+        """
+        Search through the parent directories (until root or a given cutoff directory(inclusive)) for a config file.
+        Stops at the first occurrence of the specified config file along the path to root.
+        Returns the path if a config file is found.
+        Otherwise, returns None.
+        """
+        if (local_config_root_cutoff is not None):
+            local_config_root_cutoff = os.path.abspath(local_config_root_cutoff)
+        current_directory = os.path.abspath(current_directory)
+        for _ in range(edq.util.dirent.DEPTH_LIMIT):
+            config_file_path = os.path.join(current_directory, config_file_name)
+            if (os.path.isfile(config_file_path)):
+                return config_file_path
+            # Check if current directory is root.
+            parent_dir = os.path.dirname(current_directory)
+            if (parent_dir == current_directory):
+                break
+            if (local_config_root_cutoff == current_directory):
+                break
+            current_directory = parent_dir
+        return None

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Added utility for loading tiered config from file system. #2

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!

Uh oh!