Added utility for loading tiered config from file system.

README.md

@@ -16,3 +16,95 @@ The project and Python dependencies can be installed from source with:
 ```
 pip3 install .
 ```
+
+## Configuration System
+
+This project provides a configuration system that supplies options (e.g., username, password) to a command-line interface (CLI) tool.
+The configuration system follows a tiered order, allowing options to be specified and overridden from both files and command-line options.
+
+### Configuration Sources
+
+In addition to CLI options, the configuration system loads options from [JSON](https://en.wikipedia.org/wiki/JSON) files located across multiple directories.
+By default, configuration files are named `edq-config.json`.
+This value is customizable, but this document will assume the default is used.
+
+For example, a configuration file containing the `user` and `pass` options might look like this:
+```json
+{
+    "user": "alice",
+    "pass": "password123"
+}
+```
+
+The table below summarizes the configuration sources in the order they are evaluated.
+Values from earlier sources can be overwritten by values from later sources.
+
+| Source   | Description |
+| :-----   | :---------- |
+| Global   | Loaded from a file in a user-specific location, which is platform-dependent. |
+| Local    | Loaded from a file in the current or nearest ancestor directory. |
+| CLI File | Loaded from one or more explicitly provided configuration files through the CLI. |
+| CLI      | Loaded from the command line. |
+
+The system produces an error if a global or local configuration file is unreadable (but not missing), or if a CLI-specified file is unreadable or missing.
+
+#### Global Configuration
+
+Global configuration are options that are user specific and stick with the user between projects, these are well suited for options like login credentials.
+The global configuration file defaults to `<platform-specific user configuration location>/edq-config.json`.
+The configuration location is chosen according to the [XDG standard](https://en.wikipedia.org/wiki/Freedesktop.org#Base_Directory_Specification) (implemented by [platformdirs](https://github.com/tox-dev/platformdirs)).
+Below are examples of user-specific configuration file paths for different operating systems:
+ - Linux -- `/home/<user>/.config/edq-config.json`
+ - Mac -- `/Users/<user>/Library/Application Support/edq-config.json`
+ - Windows -- `C:\Users\<user>\AppData\Local\edq-config.json`
+
+The default global configuration location can be changed by passing a path to `--config-global` through the command line.
+
+Below is an example command for specifying a global configuration path from the CLI:
+```sh
+python3 -m edq.cli.config.list --config-global  ~/.config/custom-config.json
+```
+
+#### Local Configuration
+
+Local configuration are options that are specific to a project or directory, like a project's build directory.
+Local configuration files are searched in multiple locations, the first file found is used.
+The local config search order is:
+1. `edq-config.json` in the current directory.
+2. A legacy file in the current directory (only if a legacy file is preconfigured).
+3. `edq-config.json` in any ancestor directory on the path to root (including root itself).
+
+#### CLI-Specified Config Files
+
+CLI config files are options specified on the command line via a file.
+These are useful for a common set of options you don’t need every time, such as login credentials for different user.
+Any files passed via `--config-file` will be loaded in the order they appear on the command line.
+Options from later files override options from previous files.
+
+Below is an example of a CLI specified configuration paths:
+```sh
+python3 -m edq.cli.config.list --config-file ./edq-config.json --config-file ~/.secrets/edq-config.json
+```
+
+#### CLI Configuration
+
+CLI configurations are options specified directly on the command line, these are useful for quick option overrides without editing config files.
+Configuration options are passed to the command line by the `--config` flag in this format `<key>=<value>`.
+The provided values overrides the values from configuration files.
+Configuration options are structured as key value pairs and keys cannot contain the "=" character.
+
+Below is an example of specifying a configuration option directly from the CLI:
+```sh
+python3 -m edq.cli.config.list --config user=alice --config pass=password123
+```
+
+#### CLI Config Options
+
+The table below lists common configuration CLI options available for CLI tools using this library.
+
+| CLI Option       | Description |
+| :--------------  | :---------- |
+|`--config-global` | Override the global config file location. |
+|`--config-file`   | Load configuration options from a CLI specified file. |
+| `--config`       | Provide additional options to a CLI command. |
+| `--help`         | Display standard help text and the default global configuration file path for the current platform. |

edq/core/config.py

@@ -0,0 +1,177 @@
+import argparse
+import os
+import typing
+
+import platformdirs
+
+import edq.util.dirent
+import edq.util.json
+
+CONFIG_SOURCE_GLOBAL: str = "<global config file>"
+CONFIG_SOURCE_LOCAL: str = "<local config file>"
+CONFIG_SOURCE_CLI: str = "<cli config file>"
+CONFIG_SOURCE_CLI_BARE: str = "<cli argument>"
+
+CONFIG_PATHS_KEY: str = 'config_paths'
+DEFAULT_CONFIG_FILENAME: str = "edq-config.json"
+
+class ConfigSource:
+    """ A class for storing config source information. """
+
+    def __init__(self, label: str, path: typing.Union[str, None] = None) -> None:
+        self.label = label
+        """ The label identifying the config (see CONFIG_SOURCE_* constants). """
+
+        self.path = path
+        """ The path of where the config was soruced from. """
+
+    def __eq__(self, other: object) -> bool:
+        if (not isinstance(other, ConfigSource)):
+            return False
+
+        return ((self.label == other.label) and (self.path == other.path))
+
+    def __str__(self) -> str:
+        return f"({self.label}, {self.path})"
+
+def get_tiered_config(
+        config_file_name: str = DEFAULT_CONFIG_FILENAME,
+        legacy_config_file_name: typing.Union[str, None] = None,
+        global_config_path: typing.Union[str, None] = None,
+        skip_keys: typing.Union[list, None] = None,
+        cli_arguments: typing.Union[dict, argparse.Namespace, None] = None,
+        local_config_root_cutoff: typing.Union[str, None] = None,
+    ) -> typing.Tuple[typing.Dict[str, str], typing.Dict[str, ConfigSource]]:
+    """
+    Load all configuration options from files and command-line arguments.
+    Returns a configuration dictionary with the values based on tiering rules and a source dictionary mapping each key to its origin.
+    """
+
+    if (global_config_path is None):
+        global_config_path = platformdirs.user_config_dir(config_file_name)
+
+    if (skip_keys is None):
+        skip_keys = [CONFIG_PATHS_KEY]
+
+    if (cli_arguments is None):
+        cli_arguments = {}
+
+    config: typing.Dict[str, str] = {}
+    sources: typing.Dict[str, ConfigSource] = {}
+
+    # Ensure CLI arguments are always a dict, even if provided as argparse.Namespace.
+    if (isinstance(cli_arguments, argparse.Namespace)):
+        cli_arguments = vars(cli_arguments)
+
+    # Check the global user config file.
+    if (os.path.isfile(global_config_path)):
+        _load_config_file(global_config_path, config, sources, CONFIG_SOURCE_GLOBAL)
+
+    # Check the local user config file.
+    local_config_path = _get_local_config_path(
+        config_file_name = config_file_name,
+        legacy_config_file_name = legacy_config_file_name,
+        local_config_root_cutoff = local_config_root_cutoff,
+    )
+
+    if (local_config_path is not None):
+        _load_config_file(local_config_path, config, sources, CONFIG_SOURCE_LOCAL)
+
+    # Check the config file specified on the command-line.
+    config_paths = cli_arguments.get(CONFIG_PATHS_KEY, [])
+    if (config_paths is not None):
+        for path in config_paths:
+            _load_config_file(path, config, sources, CONFIG_SOURCE_CLI)
+
+    # Finally, any command-line options.
+    for (key, value) in cli_arguments.items():
+        if (key in skip_keys):
+            continue
+
+        if ((value is None) or (value == '')):
+            continue
+
+        config[key] = value
+        sources[key] = ConfigSource(label = CONFIG_SOURCE_CLI_BARE)
+
+    return config, sources
+
+def _load_config_file(
+        config_path: str,
+        config: typing.Dict[str, str],
+        sources: typing.Dict[str, ConfigSource],
+        source_label: str,
+    ) -> None:
+    """ Loads config variables and the source from the given config JSON file. """
+
+    config_path = os.path.abspath(config_path)
+    for (key, value) in edq.util.json.load_path(config_path).items():
+        config[key] = value
+        sources[key] = ConfigSource(label = source_label, path = config_path)
+
+def _get_local_config_path(
+        config_file_name: str,
+        legacy_config_file_name: typing.Union[str, None] = None,
+        local_config_root_cutoff: typing.Union[str, None] = None,
+    ) -> typing.Union[str, None]:
+    """
+    Search for a config file in hierarchical order.
+    Begins with the provided config file name,
+    optionally checks the legacy config file name if specified,
+    then continues up the directory tree looking for the provided config file name.
+    Returns the path to the first config file found.
+
+    If no config file is found, returns None.
+
+    The cutoff parameter limits the search depth, preventing detection of config file in higher-level directories during testing.
+    """
+
+    # Provided config file is in current directory.
+    if (os.path.isfile(config_file_name)):
+        return os.path.abspath(config_file_name)
+
+    # Provided legacy config file is in current directory.
+    if (legacy_config_file_name is not None):
+        if (os.path.isfile(legacy_config_file_name)):
+            return os.path.abspath(legacy_config_file_name)
+
+    # Provided config file is found in an ancestor directory up to the root or cutoff limit.
+    parent_dir = os.path.dirname(os.getcwd())
+    return _get_ancestor_config_file_path(
+        parent_dir,
+        config_file_name = config_file_name,
+        local_config_root_cutoff = local_config_root_cutoff,
+    )
+
+def _get_ancestor_config_file_path(
+        current_directory: str,
+        config_file_name: str,
+        local_config_root_cutoff: typing.Union[str, None] = None,
+    ) -> typing.Union[str, None]:
+    """
+    Search through the parent directories (until root or a given cutoff directory(inclusive)) for a config file.
+    Stops at the first occurrence of the specified config file along the path to root.
+    Returns the path if a config file is found.
+    Otherwise, returns None.
+    """
+
+    if (local_config_root_cutoff is not None):
+        local_config_root_cutoff = os.path.abspath(local_config_root_cutoff)
+
+    current_directory = os.path.abspath(current_directory)
+    for _ in range(edq.util.dirent.DEPTH_LIMIT):
+        config_file_path = os.path.join(current_directory, config_file_name)
+        if (os.path.isfile(config_file_path)):
+            return config_file_path
+
+        # Check if current directory is root.
+        parent_dir = os.path.dirname(current_directory)
+        if (parent_dir == current_directory):
+            break
+
+        if (local_config_root_cutoff == current_directory):
+            break
+
+        current_directory = parent_dir
+
+    return None