Refactor DataPreprocessor to steps-based pipeline, add CounterDiffTransformer and quickstart config generator

[roelofsc]

Summary

• Reworks DataPreprocessor to a flexible, steps-based pipeline. This makes it easier to add new steps and data transformers to the framework.
• Adds CounterDiffTransformer for robust counter-to-diff/rate conversion (resets, rollovers, gap masking).
• Introduces generate_quickstart_config to create a ready-to-train config (and optional YAML dump).
• Added the argument features_to_select to the ColumnSelector as alternative to features_to_exclude.
• Added the argument features_to_clip to the DataClipper as alternative to features_to_exclude.
• Updates README and docs (configuration guide, examples, docstring fixes) and fixes docs build workflow (pages.yaml).
• Updates and bugfixes of the notebook Example - Hyperparameter Optimization.ipynb.

Changes

• DataPreprocessor
  • New steps-based API: DataPreprocessor(steps: Optional[List[Dict]] = None, **legacy_params).
  • STEP_REGISTRY and NAME_ALIASES to add new pipelin steps easily, with the option to add user-friendly aliases.
  • Enforced ordering:
    • NaN-introducing steps (duplicate_to_nan, counter_diff_transformer),
    • column_selector + low_unique_value_filter,
    • other transforms (angle_transformer, …),
    • simple_imputer,
    • scaler (standard_scaler/minmax_scaler; default to StandardScaler if none provided).
  • Validations:
    • Single step definitions enforced for: angle_transformer, column_selector, low_unique_value_filter, simple_imputer and scaler.
    • Auto-assign unique step names when repeated or names collide.
  • Backwards compatibility:
    • Legacy params still supported (old behavior with added counter-diff option).
    • If steps and legacy params are both given, steps take precedence with a warning.

Example config:

train:
   ....
  data_preprocessor:
    steps:
      - name: column_selector
        params:
          max_nan_frac_per_col: 0.2
      - name: low_unique_value_filter
      - name: simple_imputer
      - name: standard_scaler
    ....

• CounterDiffTransformer (new)

  • Transforms monotonic counters to per-sample increments or per-second rates.
  • Handles resets/rollovers (strategies: zero, rollover, nan, auto), small-negative tolerance, first-sample fill.
  • Optional data gap masking via time-delta threshold (factor of median dt or explicit seconds).
  • Requires DatetimeIndex when compute_rate=True or masking is enabled.

• Quickstart config generator (new)

  • generate_quickstart_config(...) builds a minimal, valid config dict and can write YAML.
  • _build_preprocessor_steps(...) composes steps for DataPreprocessor.
  • _dump_yaml_if_requested(...) safely writes YAML to disk.

• Documentation

  • README updated with a quickstart example and CLI mention.
  • New/expanded configuration guide (steps table, examples, defaults, migration notes).
  • Docstring prolems for sphinx-apidoc fixed
  • GitHub Pages generation workflow bug fixed.

[cgueck895]

I think it would be possible to define an inverse transformation by adding up the increments. Would inverse transform not be necessary for visualizing reconstructions and ARCANA-importances?

[roelofsc]

The counter resets and data gap masking loses information, which makes the simple sum of diffs inaccurate (but close?)

[roelofsc]

I have added a plotting function to plot those counter diffs/rates if needed: https://github.com/AEFDI/EnergyFaultDetector/blob/feature-counter-transformer/energy_fault_detector/utils/visualisation.py#L98

At some we probably should restructure the visualisation module to make it easiert to use... :)