County network test framework

PR_DESCRIPTION.md

@@ -0,0 +1,145 @@
+# County Network Test Framework Enhancements
+
+## Summary
+This PR enhances the county network testing framework with improved documentation, defensive attribute checking, resource monitoring, and comprehensive network inspection tools. The changes make the testing system more robust and easier to use, especially when working with networks that may have incomplete attribute initialization (e.g., OSM-derived networks).
+
+## Statistics
+- **46 files changed**: +5,645 additions, -1,267 deletions
+- **Net change**: +4,378 lines
+- **14 commits** ahead of develop
+- **No merge conflicts** ✅
+
+---
+
+## Key Changes
+
+### 1. 📚 Documentation Overhaul (~2,500 lines)
+**New comprehensive testing documentation in `docs/testing/`:**
+- **county-test-guide.md** (417 lines) - Complete guide for county-level highway testing
+- **configuration.md** (360 lines) - Detailed configuration documentation
+- **data-flow.md** (473 lines) - Visual data flow diagrams and explanations
+- **emme-manager-flow.md** (262 lines) - EMME manager interaction flows
+- **quick-start.md** (389 lines) - Quick start guide for new users
+- **network-thinning.md** (202 lines) - Network thinning strategies
+- **index.md** (87 lines) - Testing documentation index
+
+**Also added:**
+- **field-name-mapping.md** (157 lines) - CTRAMP field mapping reference
+- Updated **mkdocs.yml** with new testing section
+
+### 2. 🛡️ Defensive Attribute Checking (Production Code)
+**Enhanced `tm2py/components/network/highway/highway_network.py`** (+151/-4):
+
+Added comprehensive attribute checking in 4 methods:
+
+- **`_set_tolls()`**: Checks for `@tollbooth`, `@tollseg`, `@useclass`
+  - Gracefully skips toll processing if missing (logs warning)
+  - Useful for networks without toll facility coding
+
+- **`_set_vdf_attributes()`**: Checks for `@capclass`, `@lanes`, `@ft`, `@free_flow_speed`
+  - Raises clear KeyError if missing (critical attributes)
+
+- **`_set_link_modes()`**: Checks for `@drive_link`
+  - Raises clear KeyError if missing (critical attribute)
+
+- **`_calc_link_skim_lengths()`**: Checks for `@useclass`, `@tollbooth`
+  - Gracefully skips HOV/toll length calculations if missing
+
+**Benefits:**
+- Clear, actionable error messages instead of cryptic KeyErrors
+- Helps diagnose network attribute initialization issues
+- Backward compatible with existing networks
+- Documents required vs optional attributes
+
+### 3. 📊 Resource Monitoring
+**Enhanced `tm2py/components/network/highway/highway_assign.py`** (+88/-4):
+
+Added `monitor_resources()` context manager:
+- Monitors CPU and memory usage during SOLA assignment
+- Logs to console and separate `resource_monitor.log` file
+- Updates every 5 minutes during long-running assignments
+- Non-blocking threaded implementation
+
+**Applied to:**
+- SOLA assignment without path analysis
+- SOLA assignment with path analysis
+
+### 4. 🔧 Test Framework Improvements
+**Major refactor of `tests/run_county_test.py`** (+478/-105):
+- Better prerequisite checking
+- Enhanced logging and error handling
+- More flexible configuration system
+
+**New `tests/county_test_config.toml`** (102 lines):
+- Centralized configuration
+- Supports split EMME project and inputs sources
+
+**Enhanced `tests/highway_assign_skim_controller.py`** (+130/-2):
+- Better configuration handling
+
+### 5. 🔍 Network Inspection Tools (New Utilities)
+Created comprehensive network diagnostic tools:
+- **inspect_emme_network.py** (257 lines) - Full network inspection
+- **analyze_osm_network.py** (128 lines) - OSM network distribution analysis
+- **list_network_attributes.py** (78 lines) - Attribute lister
+- **quick_network_check.py** (76 lines) - Fast scenario checker
+- **check_modes.py** (16 lines) - Mode validation
+- **check_network_volumes.py** (53 lines) - Volume validation
+
+### 6. 📝 OSM Network Investigation
+Documentation of OSM network attribute investigation:
+- **OSM_Network_Investigation_Summary.md** (101 lines) - Investigation findings
+- **OSM_Network_Available_Attributes.md** (84 lines) - Attribute inventory
+- **osm_network_issues.md** (66 lines) - Issue tracker
+
+### 7. ⚙️ Configuration Templates
+- New **fixed_san_mateo_model.toml** (972 lines) - Complete configuration
+- New **fixed_san_mateo_scenario.toml** (76 lines) - Scenario config
+- Binary capclass files added
+- Empty placeholder files for tolls and interchange nodes
+- Removed outdated san_mateo configs
+
+### 8. 🧹 Documentation Cleanup
+Removed outdated documentation (content moved to `docs/testing/`):
+- `tests/COUNTY_TEST_FRAMEWORK_UPDATE.md` (205 lines)
+- `tests/HIGHWAY_ASSIGN_SKIM_README.md` (405 lines)
+- `tests/TESTING_INSTRUCTIONS.md` (185 lines)
+
+### 9. 🐛 Minor Fixes
+**`tm2py/emme/manager.py`** (1 line):
+- Removed `.lower()` on time_period lookup (preserves case sensitivity)
+
+---
+
+## Testing
+- ✅ Tested with 2023 San Mateo county network
+- ✅ Defensive attribute checking validated with OSM network
+- ✅ Resource monitoring tested during assignments
+- ✅ All network inspection tools executed successfully
+
+## Backward Compatibility
+- ✅ All changes are backward compatible
+- ✅ Existing networks with proper attributes work as before
+- ✅ Networks without attributes now get clear error messages instead of cryptic failures
+
+## Dependencies
+- Uses `psutil` for resource monitoring (already in requirements)
+- Uses `threading` (stdlib)
+
+---
+
+## Review Notes
+1. **Documentation**: Comprehensive testing docs now in proper location (`docs/testing/`)
+2. **Error Messages**: Much clearer for users when network attributes are missing
+3. **Debugging**: New inspection tools make network troubleshooting easier
+4. **Monitoring**: Resource usage visibility during long-running assignments
+5. **OSM Investigation**: Documents findings about OSM-derived networks lacking TM2 attributes
+
+## Questions for Reviewers
+1. Should the OSM investigation docs stay in `tests/` or move to `docs/`?
+2. Any concerns about the defensive attribute checking approach?
+3. Is the resource monitoring frequency (5 minutes) appropriate?
+
+---
+
+**Reviewer:** @lmz (Lisa Zorn)

SETUP_COMPONENT_TODO.md

@@ -0,0 +1,84 @@
+# Setup Component - Testing TODO
+
+## Status: Implementation Complete ✅, Testing In Progress
+
+### Completed:
+- ✅ Added "setup" to ComponentNames in config.py
+- ✅ Created Setup component class (components/setup.py)
+- ✅ Integrated into controller.py component_cls_map
+- ✅ Backwards compatibility verified (old configs work without changes)
+- ✅ Documentation created (docs/setup-component-integration.md)
+
+### Testing TODO:
+
+#### 1. County Network Test (IN PROGRESS)
+**Goal**: Verify setup component works with county test framework
+- [ ] Test that setup component doesn't break county test workflow
+- [ ] Verify backwards compatibility with existing county tests
+- [ ] Document any county-specific considerations
+
+#### 2. Full Model Testing (REQUIRED BEFORE MERGE)
+**Goal**: Validate setup component in production-like full model runs
+
+##### Test Scenarios:
+- [ ] **Fresh run with setup component**
+  - Create new test directory from scratch
+  - Add "setup" to initial_components in scenario_config.toml
+  - Run full model with setup as first component
+  - Verify files copied correctly
+  - Verify EMME networks initialized properly
+  - Verify subsequent components run successfully
+
+- [ ] **Existing run without setup component**
+  - Use existing model run directory (files already in place)
+  - Run model WITHOUT "setup" in initial_components
+  - Verify backwards compatibility (should work exactly as before)
+  - Confirm no errors or warnings about missing setup
+
+- [ ] **Error handling tests**
+  - Test missing setupmodel_config.toml (should error with clear message)
+  - Test invalid setupmodel_config.toml (should error with clear message)
+  - Test missing input source directories (should error with clear message)
+
+##### Test Environments:
+- [ ] Test on Windows development machine
+- [ ] Test on modeling server (if applicable)
+- [ ] Test with Box sync vs local files
+- [ ] Test with different model years (2015, 2035, etc.)
+
+##### Performance & Integration:
+- [ ] Verify setup logging works correctly
+- [ ] Check setup.log is created in run directory
+- [ ] Verify setup progress is reported in main model log
+- [ ] Test setup with warmstart enabled/disabled
+- [ ] Test setup.verify() method works correctly
+
+#### 3. Documentation Review
+- [ ] Review and update docs/setup-component-integration.md based on test results
+- [ ] Add examples from actual test runs
+- [ ] Update troubleshooting section with any issues found
+- [ ] Add performance notes (how long does setup take?)
+
+#### 4. Code Review Items
+- [ ] Review error messages for clarity
+- [ ] Check for any edge cases in file path handling
+- [ ] Verify SetupModel integration is robust
+- [ ] Consider adding progress bars for large file copies (future enhancement)
+
+### Known Issues / Considerations:
+- Setup component requires setupmodel_config.toml in run directory (not configurable path yet)
+- No dry-run mode for setup (shows what would be copied without copying)
+- File validation helper exists but not automatically called (design decision for backwards compatibility)
+
+### Before Merging:
+1. ✅ Backwards compatibility verified
+2. ⏳ County network testing complete
+3. ⏳ Full model testing complete (all scenarios above)
+4. ⏳ Documentation updated with test results
+5. ⏳ Code review complete
+6. ⏳ PR created with comprehensive description
+
+### Notes:
+- Current focus: County network test
+- After county test: Switch to full model testing
+- Keep this file updated as testing progresses

docs/assignment/field-name-mapping.md

@@ -0,0 +1,157 @@
+# Complete Field Name Mapping for TM2PY Highway Test
+
+## Critical Prerequisites
+
+### Component Execution Order
+**REQUIRED**: `create_tod_scenarios` must run before `prepare_network_highway`
+
+Why: The `create_tod_scenarios` component copies time-period-specific attributes to generic names:
+- Base scenario has: `@useclass_am`, `@useclass_ea`, `@useclass_md`, `@useclass_pm`, `@useclass_ev`
+- TOD scenarios need: `@useclass` (generic, used by highway_network.py)
+- `create_tod_scenarios` performs this copy operation
+
+Without this step, you'll get `KeyError: '@useclass'` during toll calculations.
+
+## Source Data Files
+
+### 1. Tolls CSV (E:\2015_TM2_20250619\inputs\hwy\tolls.csv)
+**Column Pattern**: `toll{period}_{vehicle}`
+```
+tollea_da, tollam_da, tollmd_da, tollpm_da, tollev_da
+tollea_s2, tollam_s2, tollmd_s2, tollpm_s2, tollev_s2
+tollea_s3, tollam_s3, tollmd_s3, tollpm_s3, tollev_s3
+tollea_vsm, tollam_vsm, tollmd_vsm, tollpm_vsm, tollev_vsm
+tollea_sml, tollam_sml, tollmd_sml, tollpm_sml, tollev_sml
+tollea_med, tollam_med, tollmd_med, tollpm_med, tollev_med
+tollea_lrg, tollam_lrg, tollmd_lrg, tollpm_lrg, tollpm_lrg, tollev_lrg
+```
+
+**Vehicle suffixes**: `da`, `s2`, `s3`, `vsm`, `sml`, `med`, `lrg`
+
+### 2. EMME Base Network Attributes
+**Time-Period-Specific Attributes** (in base scenario):
+```
+@useclass_am, @useclass_ea, @useclass_md, @useclass_pm, @useclass_ev
+```
+Values: 1=all vehicles, 2=HOV2+, 3=HOV3+, 4=autos only
+
+**Generic Attributes** (created by create_tod_scenarios in TOD scenarios):
+```
+@useclass  # Copied from @useclass_{period} for each time period scenario
+```
+
+## Code-Generated Network Attributes
+
+### From highway_network.py
+The code creates these attributes dynamically based on `dst_vehicle_group_names`:
+
+```python
+# For each vehicle in dst_vehicle_group_names:
+@bridgetoll_{vehicle}  # Bridge tolls
+@valuetoll_{vehicle}   # Value (non-bridge) tolls
+is_toll_{vehicle}      # Boolean: link has valuetoll > 0
+```
+
+## Config Requirements
+
+### highway.tolls section
+```toml
+[highway.tolls]
+file_path = "inputs/hwy/tolls.csv"
+valuetoll_start_tollbooth_code = 1000000
+src_vehicle_group_names = ["da", "s2", "s3", "vsm", "sml", "med", "lrg"]  # Must match CSV columns
+dst_vehicle_group_names = ["da", "s2", "s3", "vsm", "sml", "med", "lrg"]  # Used in network attributes
+```
+
+### highway.classes entries
+For each class, these fields must reference dst_vehicle_group_names:
+
+```toml
+veh_group_name = "da"  # Must be one of dst_vehicle_group_names
+excluded_links = ["is_toll_da"]  # Pattern: is_toll_{veh_group_name}
+toll = ["@bridgetoll_da"]  # Pattern: @bridgetoll_{veh_group_name} or @valuetoll_{veh_group_name}
+```
+
+## Validation Rules
+
+### ✓ CORRECT Naming (s2/s3 pattern)
+```toml
+[[highway.classes]]
+name = "SR2"
+veh_group_name = "s2"  # ✓ Matches dst_vehicle_group_names
+excluded_links = ["is_toll_s2"]  # ✓ Correct pattern: is_toll_s2
+toll = ["@bridgetoll_s2"]  # ✓ Correct pattern: @bridgetoll_s2
+```
+
+### ✗ INCORRECT Naming (sr2/sr3 pattern - WRONG!)
+```toml
+[[highway.classes]]
+name = "SR2"
+veh_group_name = "sr2"  # ✗ Not in dst_vehicle_group_names
+excluded_links = ["is_toll_sr2"]  # ✗ Won't be created by code
+toll = ["@bridgetoll_sr2"]  # ✗ Won't be created by code
+```
+
+## Complete Expected Config Pattern
+
+```toml
+# Tolls configuration
+[highway.tolls]
+src_vehicle_group_names = ["da", "s2", "s3", "vsm", "sml", "med", "lrg"]
+dst_vehicle_group_names = ["da", "s2", "s3", "vsm", "sml", "med", "lrg"]
+
+# DA class
+[[highway.classes]]
+name = "DA"
+veh_group_name = "da"
+excluded_links = ["is_toll_da"]
+toll = []
+
+# DA with toll
+[[highway.classes]]
+name = "DATOLL"
+veh_group_name = "da"
+excluded_links = []
+toll = ["@bridgetoll_da"]
+
+# Shared ride 2
+[[highway.classes]]
+name = "SR2"
+veh_group_name = "s2"  # ← s2, not sr2
+excluded_links = ["is_toll_s2"]  # ← is_toll_s2, not is_toll_sr2
+toll = []
+
+# Shared ride 2 with toll
+[[highway.classes]]
+name = "SR2TOLL"
+veh_group_name = "s2"  # ← s2, not sr2
+excluded_links = []
+toll = ["@bridgetoll_s2"]  # ← @bridgetoll_s2, not @bridgetoll_sr2
+
+# Shared ride 3
+[[highway.classes]]
+name = "SR3"
+veh_group_name = "s3"  # ← s3, not sr3
+excluded_links = ["is_toll_s3"]  # ← is_toll_s3, not is_toll_sr3
+toll = []
+
+# Shared ride 3 with toll
+[[highway.classes]]
+name = "SR3TOLL"
+veh_group_name = "s3"  # ← s3, not sr3
+excluded_links = []
+toll = ["@bridgetoll_s3"]  # ← @bridgetoll_s3, not @bridgetoll_sr3
+```
+
+## Summary
+
+**Key Rule**: All vehicle references must use `s2`/`s3`, NEVER `sr2`/`sr3`
+
+**Chain of consistency**:
+1. CSV columns use `s2`/`s3` → `tollam_s2`, `tollam_s3`
+2. Config src_vehicle_group_names = `["da", "s2", "s3", ...]`
+3. Config dst_vehicle_group_names = `["da", "s2", "s3", ...]`
+4. Code generates `is_toll_s2`, `@bridgetoll_s2`, `@valuetoll_s2`
+5. Config classes reference `veh_group_name = "s2"`, `excluded_links = ["is_toll_s2"]`, `toll = ["@bridgetoll_s2"]`
+
+**Any deviation breaks the chain and causes validation errors.**

docs/index.md

@@ -57,4 +57,4 @@ Important travel behavior enhancements in Travel Model Two include:
 
 ## Contributing 🎂
 
-How do you create and update these pages? See [Contributing/Documentation](contributing/documentation/)
+How do you create and update these pages? See [Contributing/Documentation](contributing/documentation.md)