[Feature Request] Implement Systematic Backtesting Engine with HMM Regime Detection

[husainm97]

Description I am looking to evolve Quant Lab Alpha from a static portfolio analyzer into a dynamic backtesting suite. The goal is to implement a Walk-Forward Analysis (WFA) framework that utilizes Hidden Markov Models (HMM) to identify market regimes (e.g., Bull, Bear, Volatile) and adjust portfolio policy accordingly.

The Vision Standard backtests often suffer from look-ahead bias and "overfitting." By implementing a Walk-Forward approach, we can optimize parameters on an in-sample "training" window and validate them on an out-of-sample "forward" window.

Proposed Workflow:

1. Regime Detection (HMM): Use hmmlearn to categorize historical data into N states based on mean returns and variance.
   

2. Modular Policy: Create a "Policy" class where users can define rules (e.g., "If HMM State = 'High Volatility', tilt 20% more toward HmL (Value) and Rm−Rf (Beta) < 0.8").
   

3. Walk-Forward Execution: * 
    -    Window 1: Train/Optimize (In-Sample).
    -    Window 2: Trade/Evaluate (Out-Sample).
    -    Shift window and repeat.
   

Technical Objectives:

1. Regime Modeling: Integration of hmmlearn or statsmodels for the Markov switching.
   

2. Backtest Engine: A vectorized or event-driven loop that tracks equity curves, drawdowns, and turnover.
   

3. Visualizations: A "Regime Map" overlaying the price chart, showing which state the HMM identified over time.
   

Requirements:

1. Must be compatible with existing FF5 and Monte Carlo modules.
   

2. Results should integrate with the (proposed) PDF Report Generator.
   

3. Clear separation between data fetching, signal generation, and performance attribution.
   

[dhruva-divate]

I think this is a substantial feature needing some plpanning as we're planning to add a complete new button in the QLA GUI. I have a proposal plan to tackle this.

Add new module structure to the root directory called /wfa (Walk-Forward Analysis) which has three main subfolders:

regime with the main module hmm_detector.py which would use hmmlearn.GaussianHMM wiyh configurable states, fit(features) to train and fit data, predict(features) method to predict and return regime levels (Bull, Bear or Volatile) and a method to return transition matrix. States ought to be sorted by mean return as expected. Features can be extracted with a features.py which supplies the input to the main module above (rolling returns, volatility and FF betas)
policy something to control the rules and adjust factor tilts. This would be intentionally lightweight at first (simple if/else style rules per regime), but extensible later into more complex optimisation logic. (e.g., "If HMM State = 'High Volatility', tilt 20% more toward HmL (Value) and Rm−Rf (Beta) < 0.8").
backtest with the actual engine to handle the walk-forward loop itself: define in-sample / out-of-sample windows, fit the regime model on the training set, apply the policy on the forward sset, then roll the window. A vectorised, event-driven loop (can be snappier then)

GUI entry point can be wfa_analysis.py in the source folder and wfa/ can have a module to orchestrate between all these listed modules

finally, from a GUI perspective, this would sit behind a single new “Walk-Forward” action, keeping the existing FF5 and Monte Carlo flows untouched.

I have opened the directory in my fork and made some small additions to gui.py as we build. In the next few days will add the basic structure for most of the modules I mentioned and update it. We should have the skeleton in a week or so.