Icf off model references

[caroarriaga]

Summary

This PR refactors the off-model calculator system to eliminate external dependencies and modernize data handling. Two major improvements: (1) variable locations now read directly from Excel files instead of external CSV, and (2) output extraction supports new vertical data format with transformation layer for pipeline compatibility.

Changes Overview

1. Variable Location Refactoring (helper/calcs.py)

Eliminated external CSV dependency - Variable locations now read directly from each calculator's Excel file.
Changes to get_variable_locations() method:

Before:

• Read from external Variable_locations.csv file
• Filtered by workbook name
• Required maintaining separate CSV file

After:

• Read from Output_test tab in calculator's own Excel file
• Self-contained, no external dependencies

2. Output Format Migration (extract_offmodel_results.py)

Added support for vertical output format with transformation layer for pipeline compatibility.

New read_output_data() function:
Purpose: Read vertical format data from Excel and transform to horizontal format expected by downstream pipeline Data Format Transformation: Old horizontal format (in Output tab):
Horizon Run ID | Out_daily_GHG_reduced_2035 | Out_per_capita_GHG_reduced_2035
2035_TM160_... | 7.100242 | -0.000096

New vertical format (in Output_test tab):
Sheet | Variable Name | Value
Main Sheet | Out_daily_GHG_reduced_2035 | 7.100242
Main Sheet | Out_per_capita_GHG_reduced_2035| -0.000096

Updated extract_off_model_calculator_result():

• Now calls read_output_data() to handle vertical format
• Currently reads from Output_test tab
• All downstream processing remains unchanged
• TODO comment added for final migration step (rename tab to Output)

Migration Notes

Pre-Deployment Requirements:

Before this PR can be merged:

1. All master calculator Excel files must have Output_test tab added
2. Output_test tab must follow vertical format structure
3. Test with actual calculator files to verify data integrity

Current State:

• Reading from Output_test tab (new vertical format)
• Old Output tab can coexist during transition
• Tab name hardcoded as 'Output_test' (line 129)

Post-Deployment Steps:

• Validate extraction works correctly with production workbook files
• After validation period, rename Output_test → Output in all Excel calculators
• Update line 129 in extract_offmodel_results.py: change output_tab_name='Output_test' to output_tab_name='Output'
• Remove old horizontal Output tabs from Excel files
• Remove Variable_locations CSV file.

[lmz]

I changed the base to master; v1.6.1_develop is going to be de deleted as it's no longer relevant (TM1.6.1 has been released) 😄