Icf off model references

utilities/RTP/Emissions/Off Model Calculators/Readme.md

@@ -1,41 +1,54 @@
-## Off-model calculation
-
-Travel Model One is not sensitive to the full range of policies MTC and ABAG may choose to pursue in RTP. Marketing and education campaigns, as well as non-capacity-increasing transportation investments likebikeshare programs, are examples of strategies with the potential to change behavior in ways that result in reduced vehicle emissions. Travel Model 1.5 and EMFAC do not estimate reductions in emissions in response to these types of changes in traveler behavior. As such, “off-model” approaches are used to quantify the GHG reduction benefits of these important climate initiatives.
-
-The off-model calculation process follows Travel Model One run. It contains the following steps, not fully automated.
-
-### Prepare model data for off-model calculation
-
-1. Run [trip-distance-by-mode-superdistrict.R](https://github.com/BayAreaMetro/travel-model-one/tree/master/utilities/bespoke-requests/trip-distance-by-mode-superdistrict) to create a super-district trip distance summary, which is not a standard model output. 
-  * run the script on modeling server where the full run data is stored:
-  ```
-  2050_TM160_DBP_Plan_04/
-    core_summaries/
-      trip-distance-by-mode-superdistrict.csv
-  ```
-  * copy the output file into M:
-  ```
-  2050_TM160_DBP_Plan_04/
-    OUTPUT/
-      bespoke/
-        trip-distance-by-mode-superdistrict.csv
-  ```
-
-2. Run the following scripts. Example call: `Rscript BikeInfrastructure.R "X:\travel-model-one-master\utilities\RTP\config_RTP2025\ModelRuns_RTP2025.xlsx" output_dir`
-  * [BikeInfrastructure.R](BikeInfrastructure.R)
-  * [Bikeshare.R](Bikeshare.R)
-  * [Carshare.R](Carshare.R)
-  * [EmployerShuttles.R](EmployerShuttles.R) (vanpools strategy)
-  * [TargetedTransportationAlternatives.R](TargetedTransportationAlternatives.R)
-
-### Create off-model calculation Excel worbooks for a given run set (including a 2035 run and a 2050 run)
-
-Start with a master version off-model calculation Excel workbook, e.g. `OffModel_Bikeshare.xlsx`.
-Run [update_offmodel_calculator_workbooks_with_TM_output.py](update_offmodel_calculator_workbooks_with_TM_output.py) with model_run_id as the arguments to create a copy of the master workbook, fill in the relevant model output data, and update the index in "Main Sheet".
-  * Example call: `python update_offmodel_calculator_workbooks.py bike_share 2035_TM160_DBP_Plan_04 2050_TM160_DBP_Plan_04`. This creates `OffModel_Bikeshare__2035_TM160_DBP_Plan_04__2050_TM160_DBP_Plan_04.xlsx`.
-
-### Manually open and save the newly created Excel workbook to trigger the calculation
-
-### Summarize off-model calculation results
-
-Once all the off-model calculation Excel workbooks for a set of model run have been created, run [summarize_offmodel_results.py](summarize_offmodel_results.py) to pull data from the Excel workbooks, summarize the results, and create summary tables, which will be automatically loaded into Travel Model Tableau dashboard ([internal link](https://10ay.online.tableau.com/#/site/metropolitantransportationcommission/views/across_RTP2025_runs/EmissionsandOff-model)). 
+# Off-model calculation
+Travel Model One is not sensitive to the full range of policies MTC and ABAG may choose to pursue in RTP. Marketing and education campaigns, as well as non-capacity-increasing transportation investments like bike share programs, are examples of strategies with the potential to change behavior in ways that result in reduced vehicle emissions. Travel Model 1.5 and EMFAC do not estimate reductions in emissions in response to these types of changes in traveler behavior. As such, “off-model” approaches are used to quantify the GHG reduction benefits of these important climate initiatives.
+
+## Current Off-Model Calculators
+In Plan Bay Area 2050, MTC/ABAG conducted off-model analysis to evaluate the GHG impacts of initiatives that could not be captured in the regional travel model. These initiatives constituted most of the key subcomponents of Strategy EN8: Expand Clean Vehicle Initiatives and Strategy EN9: Expand Transportation Demand Management Initiatives:
+
+-	Strategy EN8: Initiative EN8a – Regional Electric Vehicle Chargers
+-	Strategy EN8: Initiative EN8b – Vehicle Buyback Program 
+-	Strategy EN8: Initiative EN8c – Electric Bicycle Rebate Program - NEW
+-	Strategy EN9: Initiative EN9a – Bike Share
+-	Strategy EN9: Initiative EN9b – Car Share
+-	Strategy EN9: Initiative EN9c – Targeted Transportation Alternatives
+-	Strategy EN9: Initiative EN9d – Vanpools
+
+
+The off-model calculation process follows Travel Model One run. It contains the following steps. The process is fully automated.
+
+## Latest Updates
+- **Full automation of the process:** pulling from the RTP config files to summarizing outputs ready to be used in a Tableau dashboard.
+- **Simplification of the output tab:** the ‘Output’ tab of each calculator will no longer append a running history of the calculator ‘system state’ – only the current run – instead, this duty of storing a running history will be taken on by Tableau Online (eventually).
+- **Added to 'Variable_locations.csv':** the csv has now two columns, pointing the automated scripts to the appropriate cell locations based on whether the Travel Model run is a year 2035 or 2050 run.
+
+# General Process Workflow
+Below is the general process workflow triggered when running the main script `update_offmodel_calculator_workbooks_with_TM_output.py`:
+
+![Off-model calculator workflow](assets/off-model%20calculator%20workflow.png)
+
+## Identify model run
+The program reads the `ModelRuns_RTP2025.xlsx` file located in `utilities\RTP\config_RTP2025` and filters its data based on the condition that the column J, or Off-Model Calculator, has a "Y". 
+
+## Get model data
+The program then gets the model data belonging to the model run (e.g. 2050_TM160_DBP_Plan_04) which is required in the off-model calculator (when applicable) by creating a bespoke request.
+
+The program gets the model data from two R scripts, which run automatically:
+
+## Get SB 375 calculation
+The SB 375 tab in each calculator now lists 2005 data. There is a `SB375.csv` which holds 2005 data and will be updated manually as needed. This value is pushed automatically to each calculator.
+
+The 2005 change is only triggered by a Travel Model update, therefore, its considered a type of "off-model calculator update." When a new version of the Travel Model exists, it would recalculate 2005 and create a new version of the off-model calculators.
+
+## Calculator updates
+Each Excel Off-model calculator is updated with the following data:
+- Model run
+- Year
+- Model data
+- SB 375 data
+- Variable locations
+
+Note: the variable locations is a file that specifies all variables used in each calculator and their corresponding cell locations in Excel format (e.g. A1).
+
+Then, the results are saved in new excel workbooks.
+
+## Results from the run
+The program then stores the outputs of the Excel workbooks into a centralized logger in the server. It also creates a summarized table with the outputs of each calculator, which will be automatically loaded into the Travel Model Tableau dashboard ([internal link](https://10ay.online.tableau.com/#/site/metropolitantransportationcommission/views/across_RTP2025_runs/EmissionsandOff-model)). 

utilities/RTP/Emissions/Off Model Calculators/extract_offmodel_results.py

@@ -58,13 +58,63 @@ def get_runs_with_off_model(modelruns_xlsx):
     modelruns_with_off_model_FBP = modelruns_df.loc[modelruns_df['run_offmodel']=='FBP']
     return modelruns_with_off_model_IP, modelruns_with_off_model_FBP
 
+def read_output_data(off_model_wb, run_id, output_tab_name='Output_test'):
+    """
+    Read output data from Excel workbook.
+    Supports vertical format (Output_test/Output tab with Sheet, Variable Name, Value, Location columns).
+    Returns DataFrame in horizontal format for compatibility with existing pipeline.
+
+    Args:
+        off_model_wb: Path to Excel workbook
+        run_id: Model run ID (e.g., '2035_TM160_IPA_16')
+        output_tab_name: Name of output tab to read from (default: 'Output_test')
+
+    Returns:
+        DataFrame with columns: Horizon Run ID, Out_daily_GHG_reduced_{year}, Out_per_capita_GHG_reduced_{year}
+    """
+    year = run_id[:4]
+
+    # Read vertical format from Output tab
+    df = pd.read_excel(off_model_wb, sheet_name=output_tab_name)
+    print("Read first 15 rows from {} tab:\n{}".format(output_tab_name, df.head(15)))
+    print("")
+    # Filter for output variables (those starting with 'Out_')
+    # In the Output_test tab, output variables are identified by their variable name, not by Sheet column
+    output_vars = df[df['Variable Name'].str.startswith('Out_', na=False)].copy()
+
+    # Get year-specific variables
+    year_suffix = f'_{year}'
+    output_vars_year = output_vars[output_vars['Variable Name'].str.endswith(year_suffix)]
+
+    # Create lookup dictionary: variable_name -> value
+    var_lookup = dict(zip(output_vars_year['Variable Name'], output_vars_year['Value']))
+
+    # Extract specific variables
+    daily_ghg_key = f'Out_daily_GHG_reduced_{year}'
+    per_capita_ghg_key = f'Out_per_capita_GHG_reduced_{year}'
+
+    # Reshape to horizontal format (same structure as old Output tab)
+    result_df = pd.DataFrame({
+        'Horizon Run ID': [run_id],
+        f'Out_daily_GHG_reduced_{year}': [var_lookup.get(daily_ghg_key, None)],
+        f'Out_per_capita_GHG_reduced_{year}': [var_lookup.get(per_capita_ghg_key, None)]
+    })
+
+    # Log warnings if variables are missing
+    if daily_ghg_key not in var_lookup:
+        LOGGER.warning(f'Variable {daily_ghg_key} not found in {output_tab_name} tab')
+    if per_capita_ghg_key not in var_lookup:
+        LOGGER.warning(f'Variable {per_capita_ghg_key} not found in {output_tab_name} tab')
+
+    return result_df
+
 def extract_off_model_calculator_result(run_directory, run_id, calculator_name):
     """
     Extract the result from one off-model calculator workbook.
     """
     off_model_output_dir = os.path.join(
         run_directory, 'OUTPUT', 'offmodel', 'offmodel_output')
-    
+
     # for calculator_name in calculator_names:
     off_model_wb = os.path.join(off_model_output_dir, 'PBA50+_OffModel_{}__{}.xlsx'.format(calculator_name, run_id))
 
@@ -73,12 +123,12 @@ def extract_off_model_calculator_result(run_directory, run_id, calculator_name):
         return None
     else:
         refresh_excelworkbook(off_model_wb)
-        off_model_df = pd.read_excel(off_model_wb, sheet_name='Output', skiprows=1)
-        off_model_df = off_model_df[[
-            'Horizon Run ID', 
-            # 'Out_daily_VMT_reduced_{}'.format(run_id[:4]), 
-            'Out_daily_GHG_reduced_{}'.format(run_id[:4]), 
-            'Out_per_capita_GHG_reduced_{}'.format(run_id[:4])]]
+
+        # Read from vertical format Output_test tab (will be renamed to 'Output' after migration)
+        # TODO: Change 'Output_test' to 'Output' after Excel tab is renamed
+        off_model_df = read_output_data(off_model_wb, run_id, output_tab_name='Output_test')
+
+        # Rename columns to match expected format
         off_model_df.rename(
             columns={'Horizon Run ID': 'directory',
                      'Out_daily_GHG_reduced_{}'.format(run_id[:4]): 'daily_ghg_reduction',

utilities/RTP/Emissions/Off Model Calculators/helper/calcs.py

@@ -20,8 +20,7 @@ class OffModelCalculator:
         masterWbName: string, name of offModelCalculator of interest (e.g. bikeshare)
         dataFileName: string, name of model data file (input).
         verbose: print each method calculations.
-        varsDir: master file with all variable locations in all OffModelCalculators.
-        v: dictionary, all variable names and values for the OffModelCalculator chosen.
+        v: dictionary, all variable names and locations for the OffModelCalculator chosen.
     """
 
     def __init__(self, model_run_id, uid, verbose=False):
@@ -32,7 +31,6 @@ def __init__(self, model_run_id, uid, verbose=False):
         self.dataFileName=""
         # self.masterLogPath=common.get_master_log_path()
         self.verbose=verbose
-        self.varsDir=common.get_vars_directory()
 
     def copy_workbook(self):
         # Start run
@@ -95,17 +93,47 @@ def write_model_data_to_excel(self, data):
         OffModelCalculator.write_sbdata_to_excel(self)
 
     def get_variable_locations(self):
+        """
+        Read variable locations from the Output_test tab in the master workbook.
+        Handles both year-specific variables (e.g., variable_2035) and year-agnostic variables.
+        """
+        # Get year from run ID (first 4 characters)
+        year = self.runs[:4]
 
-        # allVars=pd.read_excel(self.varsDir)
-        allVars=pd.read_csv(self.varsDir)
-        calcVars=allVars.loc[allVars.Workbook.isin([self.masterWbName])]
-        calcVars=calcVars[['Sheet', 'Variable Name', 'Location_{}'.format(self.runs[:4])]]
-        calcVars.rename(columns={'Location_{}'.format(self.runs[:4]): 'Location'}, inplace=True)
-        groups=set(calcVars.Sheet)
-        self.v={}
+        # Read Output_test tab from master workbook
+        allVars = pd.read_excel(self.master_workbook_file, sheet_name='Output_test')
+
+        # Select relevant columns: Sheet, Variable Name, Location
+        allVars = allVars[['Sheet', 'Variable Name', 'Location']].copy()
+
+        # Split variables into two groups:
+        # 1. Variables with year suffix (e.g., variable_2035)
+        # 2. Variables without year suffix (year-agnostic)
+
+        year_suffix = f'_{year}'
+
+        # Get variables WITH year suffix for this specific year
+        vars_with_year = allVars[allVars['Variable Name'].str.endswith(year_suffix)].copy()
+        # Remove year suffix from these variables
+        vars_with_year['Variable Name'] = vars_with_year['Variable Name'].str.replace(year_suffix, '', regex=False)
+
+        # Get variables WITHOUT any year suffix (year-agnostic)
+        # A variable is year-agnostic if it doesn't end with _YYYY pattern
+        vars_without_year = allVars[~allVars['Variable Name'].str.match(r'.*_\d{4}$')].copy()
+
+        # Combine both sets of variables
+        calcVars = pd.concat([vars_with_year, vars_without_year], ignore_index=True)
+
+        # Remove duplicates (keep first occurrence)
+        calcVars = calcVars.drop_duplicates(subset=['Sheet', 'Variable Name'], keep='first')
+
+        # Group by sheet and create dictionary
+        groups = set(calcVars['Sheet'])
+        self.v = {}
         for group in groups:
-            self.v.setdefault(group,dict())
-            self.v[group]=dict(zip(calcVars['Variable Name'],calcVars['Location']))
+            self.v.setdefault(group, dict())
+            group_vars = calcVars[calcVars['Sheet'] == group]
+            self.v[group] = dict(zip(group_vars['Variable Name'], group_vars['Location']))
 
         if self.verbose:
             print("Calculator variables and locations in Excel:")

utilities/RTP/Emissions/Off Model Calculators/tests/test_config.py

@@ -0,0 +1,29 @@
+"""
+Configuration file for testing off-model calculator functions locally.
+Update these paths to point to your local test data files.
+"""
+
+import os
+
+# Base directory for test data
+TEST_DATA_DIR = os.path.join(os.path.dirname(__file__), 'test_data')
+
+# Path to Variable_locations.csv file
+# Update this to point to your local copy of the Variable_locations.csv file
+VARIABLE_LOCATIONS_CSV = os.path.join(TEST_DATA_DIR, 'Variable_locations.csv')
+
+# Example master workbook name (bike_share, car_share, etc.)
+TEST_WORKBOOK_NAME = 'PBA50+_OffModel_Bikeshare'
+
+# Example model run ID (format: YYYY_TMXXX_RunName)
+TEST_MODEL_RUN_ID = '2035_TM160_IPA_16'
+
+# Path to test Excel workbook
+# For testing with actual file, you can also use the models directory:
+# TEST_EXCEL_WORKBOOK = os.path.join(os.path.dirname(__file__), '..', 'models', f'{TEST_WORKBOOK_NAME}.xlsx')
+TEST_EXCEL_WORKBOOK = os.path.join(TEST_DATA_DIR, f'{TEST_WORKBOOK_NAME}.xlsx')
+
+# Create test_data directory if it doesn't exist
+if not os.path.exists(TEST_DATA_DIR):
+    os.makedirs(TEST_DATA_DIR)
+    print(f"Created test data directory: {TEST_DATA_DIR}")