diff --git a/utilities/RTP/Emissions/Off Model Calculators/Readme.md b/utilities/RTP/Emissions/Off Model Calculators/Readme.md index 345a4e84..a348ad87 100644 --- a/utilities/RTP/Emissions/Off Model Calculators/Readme.md +++ b/utilities/RTP/Emissions/Off Model Calculators/Readme.md @@ -1,41 +1,54 @@ -## Off-model calculation - -Travel Model One is not sensitive to the full range of policies MTC and ABAG may choose to pursue in RTP. Marketing and education campaigns, as well as non-capacity-increasing transportation investments likebikeshare programs, are examples of strategies with the potential to change behavior in ways that result in reduced vehicle emissions. Travel Model 1.5 and EMFAC do not estimate reductions in emissions in response to these types of changes in traveler behavior. As such, “off-model” approaches are used to quantify the GHG reduction benefits of these important climate initiatives. - -The off-model calculation process follows Travel Model One run. It contains the following steps, not fully automated. - -### Prepare model data for off-model calculation - -1. Run [trip-distance-by-mode-superdistrict.R](https://github.com/BayAreaMetro/travel-model-one/tree/master/utilities/bespoke-requests/trip-distance-by-mode-superdistrict) to create a super-district trip distance summary, which is not a standard model output. - * run the script on modeling server where the full run data is stored: - ``` - 2050_TM160_DBP_Plan_04/ - core_summaries/ - trip-distance-by-mode-superdistrict.csv - ``` - * copy the output file into M: - ``` - 2050_TM160_DBP_Plan_04/ - OUTPUT/ - bespoke/ - trip-distance-by-mode-superdistrict.csv - ``` - -2. Run the following scripts. Example call: `Rscript BikeInfrastructure.R "X:\travel-model-one-master\utilities\RTP\config_RTP2025\ModelRuns_RTP2025.xlsx" output_dir` - * [BikeInfrastructure.R](BikeInfrastructure.R) - * [Bikeshare.R](Bikeshare.R) - * [Carshare.R](Carshare.R) - * [EmployerShuttles.R](EmployerShuttles.R) (vanpools strategy) - * [TargetedTransportationAlternatives.R](TargetedTransportationAlternatives.R) - -### Create off-model calculation Excel worbooks for a given run set (including a 2035 run and a 2050 run) - -Start with a master version off-model calculation Excel workbook, e.g. `OffModel_Bikeshare.xlsx`. -Run [update_offmodel_calculator_workbooks_with_TM_output.py](update_offmodel_calculator_workbooks_with_TM_output.py) with model_run_id as the arguments to create a copy of the master workbook, fill in the relevant model output data, and update the index in "Main Sheet". - * Example call: `python update_offmodel_calculator_workbooks.py bike_share 2035_TM160_DBP_Plan_04 2050_TM160_DBP_Plan_04`. This creates `OffModel_Bikeshare__2035_TM160_DBP_Plan_04__2050_TM160_DBP_Plan_04.xlsx`. - -### Manually open and save the newly created Excel workbook to trigger the calculation - -### Summarize off-model calculation results - -Once all the off-model calculation Excel workbooks for a set of model run have been created, run [summarize_offmodel_results.py](summarize_offmodel_results.py) to pull data from the Excel workbooks, summarize the results, and create summary tables, which will be automatically loaded into Travel Model Tableau dashboard ([internal link](https://10ay.online.tableau.com/#/site/metropolitantransportationcommission/views/across_RTP2025_runs/EmissionsandOff-model)). +# Off-model calculation +Travel Model One is not sensitive to the full range of policies MTC and ABAG may choose to pursue in RTP. Marketing and education campaigns, as well as non-capacity-increasing transportation investments like bike share programs, are examples of strategies with the potential to change behavior in ways that result in reduced vehicle emissions. Travel Model 1.5 and EMFAC do not estimate reductions in emissions in response to these types of changes in traveler behavior. As such, “off-model” approaches are used to quantify the GHG reduction benefits of these important climate initiatives. + +## Current Off-Model Calculators +In Plan Bay Area 2050, MTC/ABAG conducted off-model analysis to evaluate the GHG impacts of initiatives that could not be captured in the regional travel model. These initiatives constituted most of the key subcomponents of Strategy EN8: Expand Clean Vehicle Initiatives and Strategy EN9: Expand Transportation Demand Management Initiatives: + +- Strategy EN8: Initiative EN8a – Regional Electric Vehicle Chargers +- Strategy EN8: Initiative EN8b – Vehicle Buyback Program +- Strategy EN8: Initiative EN8c – Electric Bicycle Rebate Program - NEW +- Strategy EN9: Initiative EN9a – Bike Share +- Strategy EN9: Initiative EN9b – Car Share +- Strategy EN9: Initiative EN9c – Targeted Transportation Alternatives +- Strategy EN9: Initiative EN9d – Vanpools + + +The off-model calculation process follows Travel Model One run. It contains the following steps. The process is fully automated. + +## Latest Updates +- **Full automation of the process:** pulling from the RTP config files to summarizing outputs ready to be used in a Tableau dashboard. +- **Simplification of the output tab:** the ‘Output’ tab of each calculator will no longer append a running history of the calculator ‘system state’ – only the current run – instead, this duty of storing a running history will be taken on by Tableau Online (eventually). +- **Added to 'Variable_locations.csv':** the csv has now two columns, pointing the automated scripts to the appropriate cell locations based on whether the Travel Model run is a year 2035 or 2050 run. + +# General Process Workflow +Below is the general process workflow triggered when running the main script `update_offmodel_calculator_workbooks_with_TM_output.py`: + +![Off-model calculator workflow](assets/off-model%20calculator%20workflow.png) + +## Identify model run +The program reads the `ModelRuns_RTP2025.xlsx` file located in `utilities\RTP\config_RTP2025` and filters its data based on the condition that the column J, or Off-Model Calculator, has a "Y". + +## Get model data +The program then gets the model data belonging to the model run (e.g. 2050_TM160_DBP_Plan_04) which is required in the off-model calculator (when applicable) by creating a bespoke request. + +The program gets the model data from two R scripts, which run automatically: + +## Get SB 375 calculation +The SB 375 tab in each calculator now lists 2005 data. There is a `SB375.csv` which holds 2005 data and will be updated manually as needed. This value is pushed automatically to each calculator. + +The 2005 change is only triggered by a Travel Model update, therefore, its considered a type of "off-model calculator update." When a new version of the Travel Model exists, it would recalculate 2005 and create a new version of the off-model calculators. + +## Calculator updates +Each Excel Off-model calculator is updated with the following data: +- Model run +- Year +- Model data +- SB 375 data +- Variable locations + +Note: the variable locations is a file that specifies all variables used in each calculator and their corresponding cell locations in Excel format (e.g. A1). + +Then, the results are saved in new excel workbooks. + +## Results from the run +The program then stores the outputs of the Excel workbooks into a centralized logger in the server. It also creates a summarized table with the outputs of each calculator, which will be automatically loaded into the Travel Model Tableau dashboard ([internal link](https://10ay.online.tableau.com/#/site/metropolitantransportationcommission/views/across_RTP2025_runs/EmissionsandOff-model)). \ No newline at end of file diff --git a/utilities/RTP/Emissions/Off Model Calculators/assets/off-model calculator workflow.png b/utilities/RTP/Emissions/Off Model Calculators/assets/off-model calculator workflow.png new file mode 100644 index 00000000..45e41301 Binary files /dev/null and b/utilities/RTP/Emissions/Off Model Calculators/assets/off-model calculator workflow.png differ diff --git a/utilities/RTP/Emissions/Off Model Calculators/extract_offmodel_results.py b/utilities/RTP/Emissions/Off Model Calculators/extract_offmodel_results.py index e3e9aea1..bf29a85f 100644 --- a/utilities/RTP/Emissions/Off Model Calculators/extract_offmodel_results.py +++ b/utilities/RTP/Emissions/Off Model Calculators/extract_offmodel_results.py @@ -58,13 +58,63 @@ def get_runs_with_off_model(modelruns_xlsx): modelruns_with_off_model_FBP = modelruns_df.loc[modelruns_df['run_offmodel']=='FBP'] return modelruns_with_off_model_IP, modelruns_with_off_model_FBP +def read_output_data(off_model_wb, run_id, output_tab_name='Output_test'): + """ + Read output data from Excel workbook. + Supports vertical format (Output_test/Output tab with Sheet, Variable Name, Value, Location columns). + Returns DataFrame in horizontal format for compatibility with existing pipeline. + + Args: + off_model_wb: Path to Excel workbook + run_id: Model run ID (e.g., '2035_TM160_IPA_16') + output_tab_name: Name of output tab to read from (default: 'Output_test') + + Returns: + DataFrame with columns: Horizon Run ID, Out_daily_GHG_reduced_{year}, Out_per_capita_GHG_reduced_{year} + """ + year = run_id[:4] + + # Read vertical format from Output tab + df = pd.read_excel(off_model_wb, sheet_name=output_tab_name) + print("Read first 15 rows from {} tab:\n{}".format(output_tab_name, df.head(15))) + print("") + # Filter for output variables (those starting with 'Out_') + # In the Output_test tab, output variables are identified by their variable name, not by Sheet column + output_vars = df[df['Variable Name'].str.startswith('Out_', na=False)].copy() + + # Get year-specific variables + year_suffix = f'_{year}' + output_vars_year = output_vars[output_vars['Variable Name'].str.endswith(year_suffix)] + + # Create lookup dictionary: variable_name -> value + var_lookup = dict(zip(output_vars_year['Variable Name'], output_vars_year['Value'])) + + # Extract specific variables + daily_ghg_key = f'Out_daily_GHG_reduced_{year}' + per_capita_ghg_key = f'Out_per_capita_GHG_reduced_{year}' + + # Reshape to horizontal format (same structure as old Output tab) + result_df = pd.DataFrame({ + 'Horizon Run ID': [run_id], + f'Out_daily_GHG_reduced_{year}': [var_lookup.get(daily_ghg_key, None)], + f'Out_per_capita_GHG_reduced_{year}': [var_lookup.get(per_capita_ghg_key, None)] + }) + + # Log warnings if variables are missing + if daily_ghg_key not in var_lookup: + LOGGER.warning(f'Variable {daily_ghg_key} not found in {output_tab_name} tab') + if per_capita_ghg_key not in var_lookup: + LOGGER.warning(f'Variable {per_capita_ghg_key} not found in {output_tab_name} tab') + + return result_df + def extract_off_model_calculator_result(run_directory, run_id, calculator_name): """ Extract the result from one off-model calculator workbook. """ off_model_output_dir = os.path.join( run_directory, 'OUTPUT', 'offmodel', 'offmodel_output') - + # for calculator_name in calculator_names: off_model_wb = os.path.join(off_model_output_dir, 'PBA50+_OffModel_{}__{}.xlsx'.format(calculator_name, run_id)) @@ -73,12 +123,12 @@ def extract_off_model_calculator_result(run_directory, run_id, calculator_name): return None else: refresh_excelworkbook(off_model_wb) - off_model_df = pd.read_excel(off_model_wb, sheet_name='Output', skiprows=1) - off_model_df = off_model_df[[ - 'Horizon Run ID', - # 'Out_daily_VMT_reduced_{}'.format(run_id[:4]), - 'Out_daily_GHG_reduced_{}'.format(run_id[:4]), - 'Out_per_capita_GHG_reduced_{}'.format(run_id[:4])]] + + # Read from vertical format Output_test tab (will be renamed to 'Output' after migration) + # TODO: Change 'Output_test' to 'Output' after Excel tab is renamed + off_model_df = read_output_data(off_model_wb, run_id, output_tab_name='Output_test') + + # Rename columns to match expected format off_model_df.rename( columns={'Horizon Run ID': 'directory', 'Out_daily_GHG_reduced_{}'.format(run_id[:4]): 'daily_ghg_reduction', diff --git a/utilities/RTP/Emissions/Off Model Calculators/helper/calcs.py b/utilities/RTP/Emissions/Off Model Calculators/helper/calcs.py index 2ccec33b..a2e248ce 100644 --- a/utilities/RTP/Emissions/Off Model Calculators/helper/calcs.py +++ b/utilities/RTP/Emissions/Off Model Calculators/helper/calcs.py @@ -20,8 +20,7 @@ class OffModelCalculator: masterWbName: string, name of offModelCalculator of interest (e.g. bikeshare) dataFileName: string, name of model data file (input). verbose: print each method calculations. - varsDir: master file with all variable locations in all OffModelCalculators. - v: dictionary, all variable names and values for the OffModelCalculator chosen. + v: dictionary, all variable names and locations for the OffModelCalculator chosen. """ def __init__(self, model_run_id, uid, verbose=False): @@ -32,7 +31,6 @@ def __init__(self, model_run_id, uid, verbose=False): self.dataFileName="" # self.masterLogPath=common.get_master_log_path() self.verbose=verbose - self.varsDir=common.get_vars_directory() def copy_workbook(self): # Start run @@ -95,17 +93,47 @@ def write_model_data_to_excel(self, data): OffModelCalculator.write_sbdata_to_excel(self) def get_variable_locations(self): + """ + Read variable locations from the Output_test tab in the master workbook. + Handles both year-specific variables (e.g., variable_2035) and year-agnostic variables. + """ + # Get year from run ID (first 4 characters) + year = self.runs[:4] - # allVars=pd.read_excel(self.varsDir) - allVars=pd.read_csv(self.varsDir) - calcVars=allVars.loc[allVars.Workbook.isin([self.masterWbName])] - calcVars=calcVars[['Sheet', 'Variable Name', 'Location_{}'.format(self.runs[:4])]] - calcVars.rename(columns={'Location_{}'.format(self.runs[:4]): 'Location'}, inplace=True) - groups=set(calcVars.Sheet) - self.v={} + # Read Output_test tab from master workbook + allVars = pd.read_excel(self.master_workbook_file, sheet_name='Output_test') + + # Select relevant columns: Sheet, Variable Name, Location + allVars = allVars[['Sheet', 'Variable Name', 'Location']].copy() + + # Split variables into two groups: + # 1. Variables with year suffix (e.g., variable_2035) + # 2. Variables without year suffix (year-agnostic) + + year_suffix = f'_{year}' + + # Get variables WITH year suffix for this specific year + vars_with_year = allVars[allVars['Variable Name'].str.endswith(year_suffix)].copy() + # Remove year suffix from these variables + vars_with_year['Variable Name'] = vars_with_year['Variable Name'].str.replace(year_suffix, '', regex=False) + + # Get variables WITHOUT any year suffix (year-agnostic) + # A variable is year-agnostic if it doesn't end with _YYYY pattern + vars_without_year = allVars[~allVars['Variable Name'].str.match(r'.*_\d{4}$')].copy() + + # Combine both sets of variables + calcVars = pd.concat([vars_with_year, vars_without_year], ignore_index=True) + + # Remove duplicates (keep first occurrence) + calcVars = calcVars.drop_duplicates(subset=['Sheet', 'Variable Name'], keep='first') + + # Group by sheet and create dictionary + groups = set(calcVars['Sheet']) + self.v = {} for group in groups: - self.v.setdefault(group,dict()) - self.v[group]=dict(zip(calcVars['Variable Name'],calcVars['Location'])) + self.v.setdefault(group, dict()) + group_vars = calcVars[calcVars['Sheet'] == group] + self.v[group] = dict(zip(group_vars['Variable Name'], group_vars['Location'])) if self.verbose: print("Calculator variables and locations in Excel:") diff --git a/utilities/RTP/Emissions/Off Model Calculators/tests/test_config.py b/utilities/RTP/Emissions/Off Model Calculators/tests/test_config.py new file mode 100644 index 00000000..a255fd77 --- /dev/null +++ b/utilities/RTP/Emissions/Off Model Calculators/tests/test_config.py @@ -0,0 +1,29 @@ +""" +Configuration file for testing off-model calculator functions locally. +Update these paths to point to your local test data files. +""" + +import os + +# Base directory for test data +TEST_DATA_DIR = os.path.join(os.path.dirname(__file__), 'test_data') + +# Path to Variable_locations.csv file +# Update this to point to your local copy of the Variable_locations.csv file +VARIABLE_LOCATIONS_CSV = os.path.join(TEST_DATA_DIR, 'Variable_locations.csv') + +# Example master workbook name (bike_share, car_share, etc.) +TEST_WORKBOOK_NAME = 'PBA50+_OffModel_Bikeshare' + +# Example model run ID (format: YYYY_TMXXX_RunName) +TEST_MODEL_RUN_ID = '2035_TM160_IPA_16' + +# Path to test Excel workbook +# For testing with actual file, you can also use the models directory: +# TEST_EXCEL_WORKBOOK = os.path.join(os.path.dirname(__file__), '..', 'models', f'{TEST_WORKBOOK_NAME}.xlsx') +TEST_EXCEL_WORKBOOK = os.path.join(TEST_DATA_DIR, f'{TEST_WORKBOOK_NAME}.xlsx') + +# Create test_data directory if it doesn't exist +if not os.path.exists(TEST_DATA_DIR): + os.makedirs(TEST_DATA_DIR) + print(f"Created test data directory: {TEST_DATA_DIR}") diff --git a/utilities/RTP/Emissions/Off Model Calculators/tests/test_data/PBA50+_OffModel_Bikeshare.xlsx b/utilities/RTP/Emissions/Off Model Calculators/tests/test_data/PBA50+_OffModel_Bikeshare.xlsx new file mode 100644 index 00000000..57fc1fb1 Binary files /dev/null and b/utilities/RTP/Emissions/Off Model Calculators/tests/test_data/PBA50+_OffModel_Bikeshare.xlsx differ diff --git a/utilities/RTP/Emissions/Off Model Calculators/tests/test_data/Variable_locations.csv b/utilities/RTP/Emissions/Off Model Calculators/tests/test_data/Variable_locations.csv new file mode 100644 index 00000000..042f3a41 --- /dev/null +++ b/utilities/RTP/Emissions/Off Model Calculators/tests/test_data/Variable_locations.csv @@ -0,0 +1,160 @@ +Workbook,Sheet,Variable Name,Description,Location_2035,Location_2050 +PBA50+_OffModel_Bikeshare,Main Sheet,VMT_displaced_conventional,VMT displaced / trip (bike + transit) - Conventional Bike,B5, +PBA50+_OffModel_Bikeshare,Main Sheet,VMT_displaced_ebike,VMT displaced / trip (bike + transit) - e-Bike ,B6, +PBA50+_OffModel_Bikeshare,Main Sheet,Total_pop_baseline,Total population,B8, +PBA50+_OffModel_Bikeshare,Main Sheet,Bikes_available_baseline,Bikes available,B9, +PBA50+_OffModel_Bikeshare,Main Sheet,Bikes_available,Bikes available,C9, +PBA50+_OffModel_Bikeshare,Main Sheet,Daily_trips_baseline,Daily trips on bike share,B10, +PBA50+_OffModel_Bikeshare,Main Sheet,Daily_trips,Daily trips on bike share,C10, +PBA50+_OffModel_Bikeshare,Main Sheet,Percent_ebikes_baseline,Percent of bikeshare that is e-bikes ,B11, +PBA50+_OffModel_Bikeshare,Main Sheet,Percent_ebikes,Percent of bikeshare that is e-bikes ,C11, +PBA50+_OffModel_Bikeshare,Main Sheet,Run_directory,Name of run directory used for 2035 data,C14,D14 +PBA50+_OffModel_Bikeshare,Main Sheet,Out_bikeshare_trips,Estimated daily bike share trips,C21, +PBA50+_OffModel_Bikeshare,Main Sheet,Out_daily_VMT_reduced,Total daily VMT reductions,C22, +PBA50+_OffModel_Bikeshare,Main Sheet,Out_daily_GHG_reduced,Total daily GHG reductions (tons),C23, +PBA50+_OffModel_Bikeshare,Main Sheet,Out_per_capita_GHG_reduced,Per capita GHG reductions,C24, +PBA50+_OffModel_Bikeshare,Main Sheet,year,,C15,D15 +PBA50+_OffModel_Carshare,Main Sheet,Min_carshare_population_density,"Minimum density needed to be considered ""urban"" and support dense car share (persons/residential acre)",B5, +PBA50+_OffModel_Carshare,Main Sheet,Participation_rate_urban,Participation of the qualifying population in 2035 - urban,C8, +PBA50+_OffModel_Carshare,Main Sheet,Participation_rate_suburban,Participation of the qualifying population in 2035 - suburban,C9, +PBA50+_OffModel_Carshare,Main Sheet,Participation_rate_NP_urban,Participation of the qualifying population in 2035 - no project; urban,C10, +PBA50+_OffModel_Carshare,Main Sheet,Participation_rate_NP_suburban,Participation of the qualifying population in 2035 - no project; suburban,C11, +PBA50+_OffModel_Carshare,Main Sheet,Daily_VMT_reduction_per_member,Daily VMT reduction per car sharing member under traditional (station-based) car sharing in 2035,C14, +PBA50+_OffModel_Carshare,Main Sheet,Daily_VMT_reduction_per_member_1way,Daily VMT reduction per car sharing member average under 1-way car sharing in 2035,D14, +PBA50+_OffModel_Carshare,Main Sheet,Travel_days,Number of travel days per year (2035),C16, +PBA50+_OffModel_Carshare,Main Sheet,TradCarshare_AnnualVMT,Annual VMT in car share vehicles by members (2035),C18, +PBA50+_OffModel_Carshare,Main Sheet,TradCarshare_AverageMPGshed,Avg mpg for cars shed/avoided (2035),C20, +PBA50+_OffModel_Carshare,Main Sheet,TradCarshare_AverageMPGcarshare,Avg mpg for car share vehicles (non-electric) (2035),C22, +PBA50+_OffModel_Carshare,Main Sheet,TradCarshare_percentEV,% of car share fleet that is battery electric (2035),C24, +PBA50+_OffModel_Carshare,Main Sheet,OneWayCarshare_AnnualVMT,Annual VMT in car share vehicles by members (2035),C27, +PBA50+_OffModel_Carshare,Main Sheet,OneWayCarshare_AverageMPGshed,Avg mpg for cars shed/avoided (2035),C29, +PBA50+_OffModel_Carshare,Main Sheet,OneWayCarshare_AverageMPGcarshare,Avg mpg for car share vehicles (non-electric) (2035),C31, +PBA50+_OffModel_Carshare,Main Sheet,OneWayCarshare_percentEV,% of 1-way car share fleet that is battery electric (2035),C33, +PBA50+_OffModel_Carshare,Main Sheet,Total_pop,Total population (2035),C39, +PBA50+_OffModel_Carshare,Main Sheet,Total_pop_hidensity,Total population in TAZs with density >10 (2035),C40, +PBA50+_OffModel_Carshare,Main Sheet,Total_pop_lodensity,Total population in TAZs with density <10 (2035),C41, +PBA50+_OffModel_Carshare,Main Sheet,Adult_pop_hidensity,Adult pop (age 20-64) in TAZs with density >10 (2035),C42, +PBA50+_OffModel_Carshare,Main Sheet,Adult_pop_lodensity,Adult pop (age 20-64) in TAZs with density <10 (2035),C43, +PBA50+_OffModel_Carshare,Main Sheet,TradCarshare_members,Number of traditional car share members (2035),C44, +PBA50+_OffModel_Carshare,Main Sheet,OneWayCarshare_members,Number of one-way car share members (2035),C45, +PBA50+_OffModel_Carshare,Main Sheet,Out_daily_VMT_reduced,Total daily VMT reductions from car sharing members driving less (2035),C47, +PBA50+_OffModel_Carshare,Main Sheet,Out_daily_GHG_reduced_VMT,Daily GHG reductions from car sharing members driving less (tons) (2035),C48, +PBA50+_OffModel_Carshare,Main Sheet,Out_daily_GHG_reduced_FE,Daily GHG reductions from car sharing members driving more fuel efficient vehicles (tons) (2035),C49, +PBA50+_OffModel_Carshare,Main Sheet,Out_daily_GHG_reduced,Total daily GHG reductions (tons) (2035),C50, +PBA50+_OffModel_Carshare,Main Sheet,Out_per_capita_GHG_reduced,Per capita GHG reductions (2035),C51, +PBA50+_OffModel_Carshare,Main Sheet,Run_directory,,C36,D36 +PBA50+_OffModel_Carshare,Main Sheet,year,,C37,D37 +PBA50+_OffModel_Carshare,Main Sheet,k_min_pop_density,,B5, +PBA50+_OffModel_Ebike,Main Sheet,Total_Funding,Total amount of funding,B5, +PBA50+_OffModel_Ebike,Main Sheet,Incentive_Amount,Incentive amount,B6, +PBA50+_OffModel_Ebike,Main Sheet,Ebikes_Incentivized,Number of e-bike purchases incentivized,B7, +PBA50+_OffModel_Ebike,Main Sheet,Start_year,Start year,B8, +PBA50+_OffModel_Ebike,Main Sheet,End_year,End year,B9, +PBA50+_OffModel_Ebike,Main Sheet,Incentive_Caused_Fraction,Fraction of incentives going to buyers who wouldn't have purchased without them,B12, +PBA50+_OffModel_Ebike,Main Sheet,Monthly_Car_Replacing_VMT,Car replacing VMT (miles per month per bike),B15, +PBA50+_OffModel_Ebike,Main Sheet,Disuse_Factor,Longterm Dis-use factor,B18, +PBA50+_OffModel_Ebike,Main Sheet,Out_daily_VMT_reduced,Total daily VMT reductions - all (2035),C24, +PBA50+_OffModel_Ebike,Main Sheet,Out_daily_GHG_reduced,Total daily GHG reductions (tons) (2035),C25, +PBA50+_OffModel_Ebike,Main Sheet,Out_per_capita_GHG_reduced,Per capita GHG reductions (2035),C26, +PBA50+_OffModel_Ebike,Main Sheet,year,,C30,D30 +PBA50+_OffModel_Ebike,Main Sheet,Run_directory,,C29,D29 +PBA50+_OffModel_RegionalCharger,Main Sheet,Num_charger_incentives,Chargers to fund (ports),B5, +PBA50+_OffModel_RegionalCharger,Main Sheet,Program_start_year,Start year of program,B6, +PBA50+_OffModel_RegionalCharger,Main Sheet,Program_end_year,End year of program,B7, +PBA50+_OffModel_RegionalCharger,Main Sheet,ACCII_toggle,,B9, +PBA50+_OffModel_RegionalCharger,Main Sheet,L2_charger_share,Share of incentivzed L2 chargers,B11, +PBA50+_OffModel_RegionalCharger,Main Sheet,DCFC_50kW_charger_share,Share of incentivzed DCFC (50 kW) chargers,B12, +PBA50+_OffModel_RegionalCharger,Main Sheet,DCFC_150kW_charger_share,Share of incentivzed DCFC (150 kW) chargers,B13, +PBA50+_OffModel_RegionalCharger,Main Sheet,DCFC_250kW_charger_share,Share of incentivzed DCFC (250 kW) chargers,B14, +PBA50+_OffModel_RegionalCharger,Main Sheet,DCFC_350kW_charger_share,Share of incentivzed DCFC (350 kW) chargers,B15, +PBA50+_OffModel_RegionalCharger,Main Sheet,L2_charger_events_daily,Average L2 charger events per day,B17, +PBA50+_OffModel_RegionalCharger,Main Sheet,DCFC_50kW_charger_events_daily,Average DCFC 50kW charger events per day,B18, +PBA50+_OffModel_RegionalCharger,Main Sheet,DCFC_150kW_charger_events_daily,Average DCFC 150kW charger events per day,B19, +PBA50+_OffModel_RegionalCharger,Main Sheet,DCFC_250kW_charger_events_daily,Average DCFC 250kW charger events per day,B20, +PBA50+_OffModel_RegionalCharger,Main Sheet,DCFC_350kW_charger_events_daily,Average DCFC 350kW charger events per day,B21, +PBA50+_OffModel_RegionalCharger,Main Sheet,L2_charger_incentive,Incentive amount for L2 chargers,B23, +PBA50+_OffModel_RegionalCharger,Main Sheet,DCFC_charger_incentive,Incentive amount for DCFC chargers,B24, +PBA50+_OffModel_RegionalCharger,Main Sheet,Out_daily_GHG_reduced,Total daily GHG reductions (tons) (2035),C34, +PBA50+_OffModel_RegionalCharger,Main Sheet,Out_per_capita_GHG_reduced,Per capita GHG reductions (2035),C35, +PBA50+_OffModel_RegionalCharger,Main Sheet,Total_program_cost,Total cumulative program cost (YOE$) (2035),C36, +PBA50+_OffModel_RegionalCharger,Main Sheet,Number_L2_incentivized,,B26, +PBA50+_OffModel_RegionalCharger,Main Sheet,Number_DCFC_50kW_incentivized,,B27, +PBA50+_OffModel_RegionalCharger,Main Sheet,Number_DCFC_150kW_incentivized,,B28, +PBA50+_OffModel_RegionalCharger,Main Sheet,Number_DCFC_250kW_incentivized,,B29, +PBA50+_OffModel_RegionalCharger,Main Sheet,Number_DCFC_350kW_incentivized,,B30, +PBA50+_OffModel_RegionalCharger,Main Sheet,year,,C33,D33 +PBA50+_OffModel_RegionalCharger,Main Sheet,Run_directory,,C32,D32 +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Run_directory,,C26,D26 +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Annual_investment_residential,Annual investment in residential programs ($2020),B5, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Annual_investment_employee,Annual investment in employee programs ($2020),B6, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Start_year,Start year of programs,B7, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Average_cost_household,Average cost/year of marketing to a household,B9, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Average_cost_employee,Average cost/year of marketing to an employee,B10, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Average_residential_penetration,Average residential penetration rate,B11, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Average_employee_penetration,Average employee penetration rate,B12, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Average_reduction_in_households,Average reduction in SOV mode share for participating households,B13, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Average_reduction_in_employees,Average reduction in SOV mode share for participating employees,B14, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Average_daily_one_way_trips_near_transit,Average daily one-way driving trips per household for households living near transit,B15, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Average_daily_commute_trips,Average daily one-way employee commute trips,B16, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Average_years_behavior_persists,Average number of years for which behavior change persists,B17, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Total_households_baseline,"Total households, 2015",B19, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Total_jobs_baseline,"Total jobs, 2015",B20, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Current_households_near_transit,Current total households living near transit stations,B22, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Current_employees_near_transit,Current total employees living near transit stations,B23, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Total_households,Total households (2035),C29, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Total_jobs,Total jobs (2035),C30, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Average_one_way_trip_length,Average one-way trip length for all trips (miles) (2035),C31, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Average_one_way_trip_length_for_home_based_work_drive_trips,Average one-way trip length for home-based work drive alone trips (miles) (2035),C32, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Total_households_that_are_likely_change,Total households that are likely to change behavior (2035),C34, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Total_employees_that_are_likely_change,Total employees that are likely to change behavior (2035),C35, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Total_households_change,Total households who change behavior (2035),C36, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Total_employees_change,Total employees who change behavior (2035),C37, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Total_daily_trip_reductions_HH,Total daily vehicle trip reductions - households (2035),C38, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Total_daily_trip_reductions_emp,Total daily vehicle trip reductions - employees (2035),C39, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Total_daily_trip_reductions,Total daily vehicle trip reductions - all (2035),C40, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Total_daily_VMT_reductions_HH,Total daily VMT reductions - households (2035),C41, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Total_daily_VMT_reductions_emp,Total daily VMT reductions - employees (2035),C42, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Out_daily_VMT_reduced,Total daily VMT reductions - all (2035),C43, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Out_daily_GHG_reduced,Total daily GHG reductions (tons) (2035),C44, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Out_per_capita_GHG_reduced,Per capita GHG reductions (2035),C45, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Total_program_cost_res,"Total cumulative program cost, residential (YOE$) (2035)",C46, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Total_program_cost_employee,"Total cumulative program cost, employee (YOE$) (2035)",C47, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,Total_program_cost,"Total cumulative program cost, all (YOE$) (2035)",C48, +PBA50+_OffModel_TargetedTransAlt,Main Sheet,year,,C27,D27 +PBA50+_OffModel_Vanpools,Main Sheet,Baseline_vanpool_vans,"Baseline number of vans, 2005",B4, +PBA50+_OffModel_Vanpools,Main Sheet,OneWay_trip_distance,One-way trip mileage (2035),C7, +PBA50+_OffModel_Vanpools,Main Sheet,Vanpool_occupancy,Average vanpool occupancy (2035),C8, +PBA50+_OffModel_Vanpools,Main Sheet,Vanpool_vans,Vanpool Program vans (2035),C9, +PBA50+_OffModel_Vanpools,Main Sheet,Vanpool_one_way_vehicle_trip_reductions,Total daily one-way vehicle trip reductions (2035),C15, +PBA50+_OffModel_Vanpools,Main Sheet,Out_daily_VMT_reduced,Total daily VMT reductions (2035),C16, +PBA50+_OffModel_Vanpools,Main Sheet,Out_daily_GHG_reduced,Total daily GHG reductions (tons) (2035),C17, +PBA50+_OffModel_Vanpools,Main Sheet,Out_per_capita_GHG_reduced,Per capita GHG reductions (2035),C18, +PBA50+_OffModel_Vanpools,Main Sheet,Run_directory,,C12,D12 +PBA50+_OffModel_Vanpools,Main Sheet,year,,C13,D13 +PBA50+_OffModel_VehicleBuyback,Main Sheet,Num_EV_incentives,Number of EVs to deploy,B5, +PBA50+_OffModel_VehicleBuyback,Main Sheet,Vehicle_replacement_age,Age of vehicle being replaced,B6, +PBA50+_OffModel_VehicleBuyback,Main Sheet,Program_start_year,Start year of program,B7, +PBA50+_OffModel_VehicleBuyback,Main Sheet,Program_end_year,End year of program,B8, +PBA50+_OffModel_VehicleBuyback,Main Sheet,Credit_sharing_toggle,Apply credit sharing adjustment,B10, +PBA50+_OffModel_VehicleBuyback,Main Sheet,PHEV_incentive_share,Share of incentivized PHEVs,B12, +PBA50+_OffModel_VehicleBuyback,Main Sheet,Average_PHEV_incentive,Average PHEV incentive,B15, +PBA50+_OffModel_VehicleBuyback,Main Sheet,Average_BEV_incentive,Average BEV incentive,B16, +PBA50+_OffModel_VehicleBuyback,Main Sheet,Out_daily_GHG_reduced,Total daily GHG reductions (tons) (2035),C23, +PBA50+_OffModel_VehicleBuyback,Main Sheet,Out_per_capita_GHG_reduced,Per capita GHG reductions (2035),C24, +PBA50+_OffModel_VehicleBuyback,Main Sheet,Total_program_cost,Total cumulative program cost (YOE$) (2035),C25, +PBA50+_OffModel_VehicleBuyback,Main Sheet,Incentive_fraction_PHEV,MPO EV Incentive Fraction for PHEV,L36, +PBA50+_OffModel_VehicleBuyback,Main Sheet,Incentive_fraction_BEV,MPO EV Incentive Fraction for BEV,M36, +PBA50+_OffModel_VehicleBuyback,Main Sheet,Rebate_count_Q1,Rebate Counts Q1,C30, +PBA50+_OffModel_VehicleBuyback,Main Sheet,Rebate_count_Q2,Rebate Counts Q2,C31, +PBA50+_OffModel_VehicleBuyback,Main Sheet,Rebate_count_Q3,Rebate Counts Q3,C32, +PBA50+_OffModel_VehicleBuyback,Main Sheet,Rebate_count_Q4,Rebate Counts Q4,C33, +PBA50+_OffModel_VehicleBuyback,Main Sheet,BEV_Rebate_amount_Q1,Rebate Amount BEV Q1,E30, +PBA50+_OffModel_VehicleBuyback,Main Sheet,BEV_Rebate_amount_Q2,Rebate Amount BEV Q2,E31, +PBA50+_OffModel_VehicleBuyback,Main Sheet,BEV_Rebate_amount_Q3,Rebate Amount BEV Q3,E32, +PBA50+_OffModel_VehicleBuyback,Main Sheet,BEV_Rebate_amount_Q4,Rebate Amount BEV Q4,E33, +PBA50+_OffModel_VehicleBuyback,Main Sheet,PHEV_Rebate_amount_Q1,Rebate Amount PHEV Q1,F30, +PBA50+_OffModel_VehicleBuyback,Main Sheet,PHEV_Rebate_amount_Q2,Rebate Amount PHEV Q2,F31, +PBA50+_OffModel_VehicleBuyback,Main Sheet,PHEV_Rebate_amount_Q3,Rebate Amount PHEV Q3,F32, +PBA50+_OffModel_VehicleBuyback,Main Sheet,PHEV_Rebate_amount_Q4,Rebate Amount PHEV Q4,F33, +PBA50+_OffModel_VehicleBuyback,Main Sheet,year,,C22,D22 +PBA50+_OffModel_VehicleBuyback,Main Sheet,Run_directory,,C21,D21 diff --git a/utilities/RTP/Emissions/Off Model Calculators/tests/test_extract_refactor.py b/utilities/RTP/Emissions/Off Model Calculators/tests/test_extract_refactor.py new file mode 100644 index 00000000..4c9ce084 --- /dev/null +++ b/utilities/RTP/Emissions/Off Model Calculators/tests/test_extract_refactor.py @@ -0,0 +1,167 @@ +""" +Test the refactored extract_offmodel_results.py with vertical format support. +""" + +import sys +import os +import pandas as pd + +# Add parent directory to path +sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..')) + +# Import the function to test +from extract_offmodel_results import read_output_data + +def test_read_output_data(): + """ + Test the read_output_data function with a sample Excel file. + """ + print("=" * 80) + print("Testing read_output_data() function") + print("=" * 80) + + # Path to test Excel file + script_dir = os.path.dirname(os.path.abspath(__file__)) + excel_file = os.path.join(script_dir, 'test_data', 'PBA50+_OffModel_Bikeshare.xlsx') + + if not os.path.exists(excel_file): + print(f"\n❌ ERROR: Test file not found: {excel_file}") + return False + + # Test parameters + run_id = '2035_TM160_IPA_16' + year = run_id[:4] + + print(f"\nTest Configuration:") + print(f" Excel File: {excel_file}") + print(f" Run ID: {run_id}") + print(f" Year: {year}") + print(f" Reading from: Output_test tab") + + try: + # Call the function + print("\n1. Calling read_output_data()...") + result_df = read_output_data(excel_file, run_id, output_tab_name='Output_test') + + print("\n2. Result DataFrame:") + print("-" * 80) + print(result_df) + print("-" * 80) + + # Validate structure + print("\n3. Validation:") + expected_columns = [ + 'Horizon Run ID', + f'Out_daily_GHG_reduced_{year}', + f'Out_per_capita_GHG_reduced_{year}' + ] + + print(f" Expected columns: {expected_columns}") + print(f" Actual columns: {list(result_df.columns)}") + + all_columns_present = all(col in result_df.columns for col in expected_columns) + + if all_columns_present: + print(" ✓ All expected columns present") + else: + print(" ❌ Missing columns!") + return False + + # Check values + print("\n4. Values:") + print(f" Horizon Run ID: {result_df['Horizon Run ID'].iloc[0]}") + print(f" Out_daily_GHG_reduced_{year}: {result_df[f'Out_daily_GHG_reduced_{year}'].iloc[0]}") + print(f" Out_per_capita_GHG_reduced_{year}: {result_df[f'Out_per_capita_GHG_reduced_{year}'].iloc[0]}") + + # Check for NaN values + has_nan = result_df.isna().any().any() + if has_nan: + print("\n ❌ ERROR: Some values are NaN (variables are missing!)") + print(f" NaN columns: {result_df.columns[result_df.isna().any()].tolist()}") + print("\n" + "=" * 80) + print("❌ TEST FAILED: Variables not found in Output_test tab") + print("=" * 80) + return False + else: + print("\n ✓ No NaN values - all variables found") + + print("\n" + "=" * 80) + print("✓ TEST PASSED: Function works correctly with vertical format") + print("=" * 80) + return True + + except Exception as e: + print(f"\n❌ ERROR: {e}") + import traceback + traceback.print_exc() + print("\n" + "=" * 80) + print("❌ TEST FAILED") + print("=" * 80) + return False + +def test_transformation(): + """ + Test that vertical → horizontal transformation works correctly. + """ + print("\n" + "=" * 80) + print("Testing Vertical → Horizontal Transformation") + print("=" * 80) + + script_dir = os.path.dirname(os.path.abspath(__file__)) + excel_file = os.path.join(script_dir, 'test_data', 'PBA50+_OffModel_Bikeshare.xlsx') + + if not os.path.exists(excel_file): + print(f"\n❌ ERROR: Test file not found") + return False + + try: + # Show vertical format + print("\n1. VERTICAL FORMAT (Input - Output_test tab):") + print("-" * 80) + df_vertical = pd.read_excel(excel_file, sheet_name='Output_test') + output_vars = df_vertical[df_vertical['Sheet'] == 'Output'] + print(output_vars[['Sheet', 'Variable Name', 'Value']].head(10).to_string()) + + # Show horizontal format + print("\n2. HORIZONTAL FORMAT (Output - after transformation):") + print("-" * 80) + df_horizontal = read_output_data(excel_file, '2035_TM160_IPA_16', output_tab_name='Output_test') + print(df_horizontal.to_string()) + + print("\n3. Transformation Summary:") + print(f" Vertical rows (Output sheet only): {len(output_vars)}") + print(f" Horizontal rows: {len(df_horizontal)}") + print(f" Horizontal columns: {len(df_horizontal.columns)}") + + print("\n✓ Transformation works - vertical data converted to horizontal format") + return True + + except Exception as e: + print(f"\n❌ ERROR: {e}") + import traceback + traceback.print_exc() + return False + +if __name__ == '__main__': + print("\nRefactored extract_offmodel_results.py Test Suite") + print("=" * 80) + + result1 = test_read_output_data() + result2 = test_transformation() + + print("\n" + "=" * 80) + print("TEST SUMMARY") + print("=" * 80) + print(f"read_output_data() test: {'✓ PASSED' if result1 else '❌ FAILED'}") + print(f"Transformation test: {'✓ PASSED' if result2 else '❌ FAILED'}") + + if result1 and result2: + print("\n✓✓✓ ALL TESTS PASSED ✓✓✓") + print("\nThe refactored extraction script is ready to use!") + print("\nNext steps:") + print("1. Test with actual workbook files") + print("2. After testing, change 'Output_test' to 'Output' in line 127") + sys.exit(0) + else: + print("\n❌❌❌ SOME TESTS FAILED ❌❌❌") + sys.exit(1) diff --git a/utilities/RTP/Emissions/Off Model Calculators/tests/test_new_variable_locations.py b/utilities/RTP/Emissions/Off Model Calculators/tests/test_new_variable_locations.py new file mode 100644 index 00000000..136033ad --- /dev/null +++ b/utilities/RTP/Emissions/Off Model Calculators/tests/test_new_variable_locations.py @@ -0,0 +1,271 @@ +""" +Test script for the updated get_variable_locations() method. +This tests reading from Output_test tab in Excel instead of Variable_locations.csv. +""" + +import sys +import os +import pandas as pd + +# Add parent directory to path to import helper modules +sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..')) + +import test_config + +def test_new_get_variable_locations(): + """ + Test the new get_variable_locations() logic that reads from Output_test tab. + This simulates what the updated method does. + """ + + print("=" * 80) + print("Testing NEW get_variable_locations() - Reading from Excel Output_test tab") + print("=" * 80) + + excel_file = test_config.TEST_EXCEL_WORKBOOK + runs = test_config.TEST_MODEL_RUN_ID + + # Check if Excel file exists + if not os.path.exists(excel_file): + print(f"\nERROR: Excel workbook not found at:") + print(f" {excel_file}") + print("\nPlease update test_config.TEST_EXCEL_WORKBOOK with path to your Excel file.") + return None + + print(f"\n1. Reading from Excel workbook:") + print(f" File: {excel_file}") + print(f" Sheet: Output_test") + print(f" Model Run: {runs}") + + # Get year from run ID (first 4 characters) + year = runs[:4] + print(f" Year extracted: {year}") + + try: + # Read Output_test tab from Excel + allVars = pd.read_excel(excel_file, sheet_name='Output_test') + print(f"\n2. Total rows in Output_test: {len(allVars)}") + print(f" Columns: {list(allVars.columns)}") + + # Show sample of all data + print(f"\n3. Sample of Output_test (first 10 rows):") + print(allVars.head(10).to_string()) + + # Select relevant columns first + allVars = allVars[['Sheet', 'Variable Name', 'Location']].copy() + + # Split variables into two groups: + # 1. Variables with year suffix (e.g., variable_2035) + # 2. Variables without year suffix (year-agnostic) + + year_suffix = f'_{year}' + print(f"\n4. Processing variables:") + print(f" Year suffix to filter: '{year_suffix}'") + + # Get variables WITH year suffix for this specific year + vars_with_year = allVars[allVars['Variable Name'].str.endswith(year_suffix)].copy() + print(f" Variables WITH year suffix ({year}): {len(vars_with_year)}") + + # Get variables WITHOUT any year suffix (year-agnostic) + # A variable is year-agnostic if it doesn't end with _YYYY pattern + vars_without_year = allVars[~allVars['Variable Name'].str.match(r'.*_\d{4}$')].copy() + print(f" Variables WITHOUT year suffix (year-agnostic): {len(vars_without_year)}") + + if len(vars_with_year) == 0 and len(vars_without_year) == 0: + print(f"\n WARNING: No variables found!") + return None + + print(f"\n5. Variables WITH year suffix (before removing suffix):") + print(vars_with_year.head(10).to_string()) + + # Remove year suffix from year-specific variables + vars_with_year['Variable Name'] = vars_with_year['Variable Name'].str.replace(year_suffix, '', regex=False) + + print(f"\n6. Variables WITHOUT year suffix (year-agnostic):") + print(vars_without_year.to_string()) + + # Combine both sets of variables + calcVars = pd.concat([vars_with_year, vars_without_year], ignore_index=True) + + print(f"\n7. Combined variables (total: {len(calcVars)}):") + print(calcVars.to_string()) + + # Check for duplicates + duplicates = calcVars[calcVars.duplicated(subset=['Sheet', 'Variable Name'], keep=False)] + if len(duplicates) > 0: + print(f"\n ⚠ WARNING: Found {len(duplicates)} duplicate variable entries:") + print(duplicates.sort_values(['Sheet', 'Variable Name']).to_string()) + print("\n Removing duplicates (keeping first occurrence)...") + calcVars = calcVars.drop_duplicates(subset=['Sheet', 'Variable Name'], keep='first') + + # Group by sheet + groups = set(calcVars['Sheet']) + print(f"\n8. Sheets found: {groups} (after deduplication)") + + # Create dictionary structure + v = {} + for group in groups: + v.setdefault(group, dict()) + group_vars = calcVars[calcVars['Sheet'] == group] + v[group] = dict(zip(group_vars['Variable Name'], group_vars['Location'])) + + print(f"\n9. Final dictionary structure (v):") + print("-" * 80) + for sheet, variables in v.items(): + print(f"\nSheet: '{sheet}'") + for var_name, location in variables.items(): + print(f" {var_name:40s} -> {location}") + + print("\n" + "=" * 80) + print("Test completed successfully!") + print("=" * 80) + print("\nKey changes from CSV method:") + print(" ✓ Reads from Excel Output_test tab (not external CSV)") + print(" ✓ Filters by year suffix in variable name (not column name)") + print(" ✓ Removes year suffix for clean variable names") + print(" ✓ No dependency on external Variable_locations.csv file") + + return v + + except Exception as e: + print(f"\n❌ ERROR: {str(e)}") + import traceback + traceback.print_exc() + return None + + +def compare_csv_vs_excel(): + """ + Compare results from old CSV method vs new Excel method. + """ + print("\n" + "=" * 80) + print("COMPARISON: CSV method vs Excel method") + print("=" * 80) + + # Test old method (CSV) + csv_path = test_config.VARIABLE_LOCATIONS_CSV + excel_path = test_config.TEST_EXCEL_WORKBOOK + runs = test_config.TEST_MODEL_RUN_ID + workbook_name = test_config.TEST_WORKBOOK_NAME + year = runs[:4] + + csv_result = None + excel_result = None + + # Try old CSV method + if os.path.exists(csv_path): + print("\n1. OLD METHOD (CSV):") + print(f" Reading from: {csv_path}") + try: + allVars = pd.read_csv(csv_path) + calcVars = allVars.loc[allVars['Workbook'].isin([workbook_name])] + location_col = f'Location_{year}' + if location_col in calcVars.columns: + calcVars = calcVars[['Sheet', 'Variable Name', location_col]].copy() + calcVars.rename(columns={location_col: 'Location'}, inplace=True) + + csv_result = {} + for sheet in set(calcVars['Sheet']): + sheet_vars = calcVars[calcVars['Sheet'] == sheet] + csv_result[sheet] = dict(zip(sheet_vars['Variable Name'], sheet_vars['Location'])) + + print(f" ✓ Found {len(calcVars)} variables") + else: + print(f" ✗ Column '{location_col}' not found") + except Exception as e: + print(f" ✗ Error: {e}") + else: + print(f"\n1. OLD METHOD (CSV): Skipped (file not found)") + + # Try new Excel method + if os.path.exists(excel_path): + print("\n2. NEW METHOD (Excel):") + print(f" Reading from: {excel_path} -> Output_test") + try: + allVars = pd.read_excel(excel_path, sheet_name='Output_test') + allVars = allVars[['Sheet', 'Variable Name', 'Location']].copy() + + year_suffix = f'_{year}' + + # Get variables WITH year suffix for this specific year + vars_with_year = allVars[allVars['Variable Name'].str.endswith(year_suffix)].copy() + vars_with_year['Variable Name'] = vars_with_year['Variable Name'].str.replace(year_suffix, '', regex=False) + + # Get variables WITHOUT any year suffix (year-agnostic) + vars_without_year = allVars[~allVars['Variable Name'].str.match(r'.*_\d{4}$')].copy() + + # Combine both sets + calcVars = pd.concat([vars_with_year, vars_without_year], ignore_index=True) + + # Remove duplicates (keep first occurrence) + calcVars = calcVars.drop_duplicates(subset=['Sheet', 'Variable Name'], keep='first') + + excel_result = {} + for sheet in set(calcVars['Sheet']): + sheet_vars = calcVars[calcVars['Sheet'] == sheet] + excel_result[sheet] = dict(zip(sheet_vars['Variable Name'], sheet_vars['Location'])) + + print(f" ✓ Found {len(calcVars)} unique variables ({len(vars_with_year)} with year, {len(vars_without_year)} year-agnostic)") + except Exception as e: + print(f" ✗ Error: {e}") + else: + print(f"\n2. NEW METHOD (Excel): Skipped (file not found)") + + # Compare + if csv_result and excel_result: + print("\n3. COMPARISON:") + + # Get all unique sheet names + all_sheets = set(list(csv_result.keys()) + list(excel_result.keys())) + + for sheet_name in sorted(all_sheets): + csv_vars = csv_result.get(sheet_name, {}) + excel_vars = excel_result.get(sheet_name, {}) + + all_var_names = set(list(csv_vars.keys()) + list(excel_vars.keys())) + + print(f"\n Sheet: '{sheet_name}'") + print(f" {'Variable Name':40s} {'CSV Location':15s} {'Excel Location':15s} {'Match':10s}") + print(f" {'-'*80}") + + for var_name in sorted(all_var_names): + csv_loc = csv_vars.get(var_name, 'N/A') + excel_loc = excel_vars.get(var_name, 'N/A') + match = '✓' if csv_loc == excel_loc else '✗' + print(f" {var_name:40s} {csv_loc:15s} {excel_loc:15s} {match:10s}") + + # Print summary + print("\n" + "=" * 80) + print("SUMMARY:") + total_matches = 0 + total_vars = 0 + for sheet_name in all_sheets: + csv_vars = csv_result.get(sheet_name, {}) + excel_vars = excel_result.get(sheet_name, {}) + all_var_names = set(list(csv_vars.keys()) + list(excel_vars.keys())) + + for var_name in all_var_names: + total_vars += 1 + if csv_vars.get(var_name) == excel_vars.get(var_name) and csv_vars.get(var_name) != None: + total_matches += 1 + + print(f"Matching variables: {total_matches}/{total_vars}") + print("=" * 80) + + +if __name__ == '__main__': + print("\nUpdated get_variable_locations() Test") + print("=" * 80) + print(f"Test configuration:") + print(f" Workbook: {test_config.TEST_WORKBOOK_NAME}") + print(f" Model Run: {test_config.TEST_MODEL_RUN_ID}") + print(f" Excel Path: {test_config.TEST_EXCEL_WORKBOOK}") + print("=" * 80) + + # Run main test + result = test_new_get_variable_locations() + + # Run comparison if both files exist + if os.path.exists(test_config.VARIABLE_LOCATIONS_CSV) and os.path.exists(test_config.TEST_EXCEL_WORKBOOK): + print("\n") + compare_csv_vs_excel() diff --git a/utilities/RTP/Emissions/Off Model Calculators/tests/test_variable_locations.py b/utilities/RTP/Emissions/Off Model Calculators/tests/test_variable_locations.py new file mode 100644 index 00000000..f0101aa4 --- /dev/null +++ b/utilities/RTP/Emissions/Off Model Calculators/tests/test_variable_locations.py @@ -0,0 +1,204 @@ +""" +Test script to understand how get_variable_locations() works. +This script isolates and tests the get_variable_locations() function +without needing access to all production paths. +""" + +import sys +import os +import pandas as pd + +# Add parent directory to path to import helper modules +sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..')) + +import test_config + +def test_get_variable_locations(): + """ + Mock version of get_variable_locations() from calcs.py + This demonstrates how the function reads and processes Variable_locations.csv + """ + + print("=" * 80) + print("Testing get_variable_locations() function") + print("=" * 80) + + # Check if Variable_locations.csv exists + if not os.path.exists(test_config.VARIABLE_LOCATIONS_CSV): + print(f"\nERROR: Variable_locations.csv not found at:") + print(f" {test_config.VARIABLE_LOCATIONS_CSV}") + print("\nPlease create a test_data folder and add your Variable_locations.csv file.") + print(f"Expected location: {test_config.TEST_DATA_DIR}") + return None + + print(f"\n1. Reading Variable_locations.csv from:") + print(f" {test_config.VARIABLE_LOCATIONS_CSV}") + + # Read the CSV file + allVars = pd.read_csv(test_config.VARIABLE_LOCATIONS_CSV) + + print(f"\n2. Total variables in CSV: {len(allVars)}") + print(f" Columns: {list(allVars.columns)}") + + # Show sample of all data + print(f"\n3. Sample of all variables (first 10 rows):") + print(allVars.head(10).to_string()) + + # Filter by workbook name + masterWbName = test_config.TEST_WORKBOOK_NAME + runs = test_config.TEST_MODEL_RUN_ID + year = runs[:4] # Extract year from run ID (e.g., "2035" from "2035_TM160_IPA_16") + + print(f"\n4. Filtering for workbook: '{masterWbName}'") + print(f" Model run: {runs}") + print(f" Year: {year}") + + calcVars = allVars.loc[allVars['Workbook'].isin([masterWbName])] + + print(f"\n5. Variables found for this workbook: {len(calcVars)}") + + if len(calcVars) == 0: + print(f"\n WARNING: No variables found for workbook '{masterWbName}'") + print(f" Available workbooks in CSV:") + if 'Workbook' in allVars.columns: + for wb in allVars['Workbook'].unique(): + print(f" - {wb}") + return None + + # Select relevant columns + location_col = f'Location_{year}' + + print(f"\n6. Looking for location column: '{location_col}'") + + if location_col not in calcVars.columns: + print(f"\n WARNING: Column '{location_col}' not found!") + print(f" Available location columns:") + location_cols = [col for col in calcVars.columns if col.startswith('Location_')] + for col in location_cols: + print(f" - {col}") + + if location_cols: + location_col = location_cols[0] + print(f"\n Using first available column: '{location_col}'") + else: + print("\n No location columns found!") + return None + + # Select and rename columns + calcVars = calcVars[['Sheet', 'Variable Name', location_col]].copy() + calcVars.rename(columns={location_col: 'Location'}, inplace=True) + + print(f"\n7. Processed variables (all rows):") + print(calcVars.to_string()) + + # Group by sheet + groups = set(calcVars['Sheet']) + print(f"\n8. Sheets found: {groups}") + + # Create dictionary structure + v = {} + for group in groups: + v.setdefault(group, dict()) + group_vars = calcVars[calcVars['Sheet'] == group] + v[group] = dict(zip(group_vars['Variable Name'], group_vars['Location'])) + + print(f"\n9. Final dictionary structure (v):") + print("-" * 80) + for sheet, variables in v.items(): + print(f"\nSheet: '{sheet}'") + for var_name, location in variables.items(): + print(f" {var_name:30s} -> {location}") + + print("\n" + "=" * 80) + print("Test completed successfully!") + print("=" * 80) + + return v + + +def create_sample_csv(): + """ + Create a sample Variable_locations.csv file for testing. + This shows the expected structure of the file. + """ + sample_data = { + 'Workbook': [ + 'PBA50+_OffModel_Bikeshare', + 'PBA50+_OffModel_Bikeshare', + 'PBA50+_OffModel_Bikeshare', + 'PBA50+_OffModel_Carshare', + 'PBA50+_OffModel_Carshare', + ], + 'Sheet': [ + 'Main Sheet', + 'Main Sheet', + 'Calculations', + 'Main Sheet', + 'Main Sheet', + ], + 'Variable Name': [ + 'Run_directory', + 'year', + 'total_trips', + 'Run_directory', + 'year', + ], + 'Location_2035': [ + 'B5', + 'B6', + 'C10', + 'B5', + 'B6', + ], + 'Location_2050': [ + 'B5', + 'B6', + 'C10', + 'B5', + 'B6', + ] + } + + df = pd.DataFrame(sample_data) + sample_path = os.path.join(test_config.TEST_DATA_DIR, 'Variable_locations_SAMPLE.csv') + df.to_csv(sample_path, index=False) + + print(f"\nSample Variable_locations.csv created at:") + print(f" {sample_path}") + print("\nSample structure:") + print(df.to_string()) + print(f"\nRename this file to 'Variable_locations.csv' to use it for testing.") + + +if __name__ == '__main__': + print("\nOff-Model Calculator Variable Locations Test") + print("=" * 80) + print(f"Test configuration:") + print(f" Workbook: {test_config.TEST_WORKBOOK_NAME}") + print(f" Model Run: {test_config.TEST_MODEL_RUN_ID}") + print(f" CSV Path: {test_config.VARIABLE_LOCATIONS_CSV}") + print("=" * 80) + + # Check if Variable_locations.csv exists + if not os.path.exists(test_config.VARIABLE_LOCATIONS_CSV): + print("\n[INFO] Variable_locations.csv not found. Creating sample file...") + create_sample_csv() + print("\n[ACTION REQUIRED]") + print("1. Check the test_data folder") + print("2. Either:") + print(" a) Copy your actual Variable_locations.csv to test_data/, OR") + print(" b) Rename Variable_locations_SAMPLE.csv to Variable_locations.csv") + print("3. Update test_config.py with correct workbook name and run ID") + print("4. Run this script again") + else: + # Run the test + result = test_get_variable_locations() + + if result: + print("\n[SUCCESS] Variable locations loaded successfully!") + print("\nYou can now understand how the data flows:") + print("1. CSV file contains mappings for all calculators") + print("2. Filtered by workbook name (calculator type)") + print("3. Organized by sheet name") + print("4. Each variable maps to an Excel cell location") + print("5. Used by write_runid_to_mainsheet() to write data to Excel") diff --git a/utilities/RTP/Emissions/Off Model Calculators/tests/validate_output_tabs.py b/utilities/RTP/Emissions/Off Model Calculators/tests/validate_output_tabs.py new file mode 100644 index 00000000..28b80a3a --- /dev/null +++ b/utilities/RTP/Emissions/Off Model Calculators/tests/validate_output_tabs.py @@ -0,0 +1,207 @@ +""" +Validation script to compare Output tab (old horizontal format) vs Output_test tab (new vertical format). +Ensures no variables are lost in the transition. +""" + +import pandas as pd +import os +import sys + +def get_variables_from_output_tab(excel_file, year): + """ + Extract variable names from old Output tab (horizontal format). + Returns set of variable names. + """ + try: + # Read Output tab (skip first row which is usually headers) + df = pd.read_excel(excel_file, sheet_name='Output', skiprows=1) + + # Get column names (these are the variables in horizontal format) + variables = set(df.columns) + + # Filter for year-specific variables + year_variables = {col for col in variables if str(year) in str(col)} + + return variables, year_variables + except Exception as e: + print(f"Error reading Output tab: {e}") + return set(), set() + +def get_variables_from_output_test_tab(excel_file, year): + """ + Extract variable names from new Output_test tab (vertical format). + Returns set of variable names. + """ + try: + # Read Output_test tab + df = pd.read_excel(excel_file, sheet_name='Output_test') + + # Filter for Output sheet + output_vars = df[df['Sheet'] == 'Output'].copy() + + # Get all variable names + all_variables = set(output_vars['Variable Name'].dropna()) + + # Get year-specific variables (ending with _YYYY) + year_suffix = f'_{year}' + year_variables = {var for var in all_variables if var.endswith(year_suffix)} + + # Get year-agnostic variables (not ending with _YYYY) + agnostic_variables = {var for var in all_variables if not var[-5:].startswith('_') or not var[-4:].isdigit()} + + return all_variables, year_variables, agnostic_variables + except Exception as e: + print(f"Error reading Output_test tab: {e}") + return set(), set(), set() + +def compare_output_tabs(excel_file, year): + """ + Compare Output and Output_test tabs to ensure no variables are lost. + """ + print("=" * 80) + print(f"Validating Output Tabs for Year {year}") + print("=" * 80) + print(f"File: {excel_file}\n") + + # Get variables from old Output tab + print("1. Reading Output tab (horizontal format)...") + output_all, output_year = get_variables_from_output_tab(excel_file, year) + print(f" Total columns: {len(output_all)}") + print(f" Year-specific variables ({year}): {len(output_year)}") + + # Get variables from new Output_test tab + print("\n2. Reading Output_test tab (vertical format)...") + test_all, test_year, test_agnostic = get_variables_from_output_test_tab(excel_file, year) + print(f" Total variables: {len(test_all)}") + print(f" Year-specific variables ({year}): {len(test_year)}") + print(f" Year-agnostic variables: {len(test_agnostic)}") + + # Remove year suffix from Output_test variables for comparison + test_year_normalized = {var.replace(f'_{year}', '') for var in test_year} + + # Compare + print("\n3. COMPARISON:") + print("-" * 80) + + # Show Output tab variables + print(f"\nOutput tab variables (year {year}):") + for var in sorted(output_year): + print(f" - {var}") + + # Show Output_test tab year-specific variables (before normalization) + print(f"\nOutput_test tab year-specific variables (year {year}):") + for var in sorted(test_year): + print(f" - {var}") + + # Show Output_test tab year-agnostic variables + if test_agnostic: + print(f"\nOutput_test tab year-agnostic variables:") + for var in sorted(test_agnostic): + print(f" - {var}") + + # Check for missing variables + print("\n4. VALIDATION:") + print("-" * 80) + + # Normalize Output tab variable names (remove year suffix if present) + output_year_normalized = {var.replace(f'_{year}', '') for var in output_year if f'_{year}' in var} + + # Variables in Output but not in Output_test + missing_in_test = output_year_normalized - test_year_normalized - test_agnostic + + # Variables in Output_test but not in Output + new_in_test = test_year_normalized - output_year_normalized + + if missing_in_test: + print(f"\n❌ MISSING VARIABLES in Output_test (present in Output):") + for var in sorted(missing_in_test): + print(f" - {var}") + else: + print(f"\n✓ All Output tab variables are present in Output_test") + + if new_in_test: + print(f"\n✓ NEW VARIABLES in Output_test (not in Output):") + for var in sorted(new_in_test): + print(f" - {var}") + + # Summary + print("\n5. SUMMARY:") + print("=" * 80) + print(f"Output tab variables: {len(output_year_normalized)}") + print(f"Output_test year-specific: {len(test_year_normalized)}") + print(f"Output_test year-agnostic: {len(test_agnostic)}") + print(f"Output_test total (normalized): {len(test_year_normalized) + len(test_agnostic)}") + print(f"Missing variables: {len(missing_in_test)}") + print(f"New variables: {len(new_in_test)}") + + if missing_in_test: + print("\n❌ VALIDATION FAILED: Some variables are missing!") + return False + else: + print("\n✓ VALIDATION PASSED: All variables accounted for!") + return True + +def validate_specific_extraction_variables(excel_file, year): + """ + Validate that the specific variables needed by extract_offmodel_results.py are present. + """ + print("\n" + "=" * 80) + print("Validating Required Extraction Variables") + print("=" * 80) + + required_vars = [ + f'Out_daily_GHG_reduced_{year}', + f'Out_per_capita_GHG_reduced_{year}' + ] + + try: + df = pd.read_excel(excel_file, sheet_name='Output_test') + output_vars = df[df['Sheet'] == 'Output'] + available_vars = set(output_vars['Variable Name'].dropna()) + + print("\nRequired variables for extraction:") + all_present = True + for var in required_vars: + if var in available_vars: + print(f" ✓ {var}") + else: + print(f" ❌ {var} - MISSING!") + all_present = False + + if all_present: + print("\n✓ All required variables are present") + return True + else: + print("\n❌ Some required variables are missing!") + return False + + except Exception as e: + print(f"\n❌ Error: {e}") + return False + +if __name__ == '__main__': + # Path to test file + script_dir = os.path.dirname(os.path.abspath(__file__)) + excel_file = os.path.join(script_dir, 'test_data', 'PBA50+_OffModel_Bikeshare.xlsx') + + if not os.path.exists(excel_file): + print(f"ERROR: Excel file not found: {excel_file}") + sys.exit(1) + + # Test for year 2035 (adjust as needed) + year = '2035' + + # Run comparisons + result1 = compare_output_tabs(excel_file, year) + result2 = validate_specific_extraction_variables(excel_file, year) + + if result1 and result2: + print("\n" + "=" * 80) + print("✓✓✓ ALL VALIDATIONS PASSED ✓✓✓") + print("=" * 80) + sys.exit(0) + else: + print("\n" + "=" * 80) + print("❌❌❌ VALIDATION FAILED ❌❌❌") + print("=" * 80) + sys.exit(1) diff --git a/utilities/RTP/Emissions/Off Model Calculators/tests/verify_excel_structure.py b/utilities/RTP/Emissions/Off Model Calculators/tests/verify_excel_structure.py new file mode 100644 index 00000000..44e5bfcd --- /dev/null +++ b/utilities/RTP/Emissions/Off Model Calculators/tests/verify_excel_structure.py @@ -0,0 +1,50 @@ +""" +Quick script to verify Excel file has the Output_test tab with expected structure. +""" + +import pandas as pd +import os + +# Path to Excel file (relative to this script's location) +script_dir = os.path.dirname(os.path.abspath(__file__)) +excel_file = os.path.join(script_dir, 'test_data', 'PBA50+_OffModel_Bikeshare.xlsx') + +if not os.path.exists(excel_file): + print(f"ERROR: File not found: {excel_file}") +else: + print(f"Checking Excel file: {excel_file}\n") + + # Read Excel file to get sheet names + xl = pd.ExcelFile(excel_file) + print(f"Available sheets:") + for i, sheet in enumerate(xl.sheet_names, 1): + print(f" {i}. {sheet}") + + # Check if Output_test exists + if 'Output_test' in xl.sheet_names: + print(f"\n✓ Output_test sheet found!") + + # Read Output_test tab + df = pd.read_excel(excel_file, sheet_name='Output_test') + + print(f"\nOutput_test structure:") + print(f" Rows: {len(df)}") + print(f" Columns: {list(df.columns)}") + + print(f"\nFirst 15 rows:") + print(df.head(15).to_string()) + + # Check for year suffixes + print(f"\nVariable names with year suffixes:") + year_vars = df[df['Variable Name'].str.contains('_20', na=False)] + if len(year_vars) > 0: + unique_vars = year_vars['Variable Name'].unique()[:10] + for var in unique_vars: + print(f" - {var}") + print(f" ... (showing first 10 of {len(year_vars)} variables)") + else: + print(" No variables with year suffixes found") + + else: + print(f"\n✗ Output_test sheet NOT found!") + print(f"Please add Output_test sheet to the Excel file.")