Bring BayAreaMetro/network_wrangler up to date with changes from wsp-sag/network_wrangler develop #3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
when `managed` exists in the network (before applying the managed project card), and `managed` is not specified for newly added links (which can be assumed 0).
cherry pick generic fixes from MTC to `develop`
Hotfix: ability to select all links in a project card
Added flake config in setup.cfg
# Fixes several bugs that are mainly due to evolved APIs for various dependences ## 1. OSMNX v>1.0 Issue: `osmnx.graph_from_gdfs()` API changed to require the edge dataframe should have u,v,key as a multi-index - but earlier versions require that it isn't. Resolution: query the version of OSMNX before making the appropriate method call ## Folium Issue: error due to missing geom arg https://osmnx.readthedocs.io/en/latest/internals.html#osmnx-folium-module ## PyYaml * Use yaml.safe_load() since yaml.load() now requires a Loader argument https://pyyaml.org/wiki/PyYAMLDocumentation # Other Bugs ## `test_add_adhoc_managed_lane_field()` - It was trying to do a selection with Select Key: (model_link_id=="224") - Managed lanes require information about access/egress Resolved confusing distinction between "access" the field denoting who can be on the link and "access links" and "access points". ## folium - Missing imports (osmnx, folium) - Missing geom arg to osmx.folium._make_folium_polyline() - Use USERPROFILE/HOME to determine OUTPATH ## types * wheelchair_accessible is a float not a string * It's an enum, and can be blank, 0, 1 or 2 https://developers.google.com/transit/gtfs/reference#tripstxt # Chores * pep8 / flaked / blacked * Added flake config in setup.cfg Closes #257, #280 Co-authored-by: Elizabeth Sall <easall@gmail.com>
- Added a basic pytest gh action set to run on push events - Updated some of the basic documentation about running tests and documentation to match - Separate requirements into different files so don't need to install all potentially messy requirements if you don't need them - Each requirements file is now an install_extra in setuptools so can be installed when installing network wrangler like pip install .[tests,docs,viz] etc. - Remove travis pytest.marks and travis.yml and other travis remnants
Added Functionality: * updates link, shape, and node geometry when node x or y is changed - added step to make sure new nodes had correct geometry - added better testing to make sure it was 'safe' to delete nodes - Added RoadwayNetwork class variables for properties to coerce - Added class function to do coercion and call it at read-in and write out - convenience method added to RoadwayNetwork to count number of managed lane links Lots of refactoring: - refactored how to delete nodes and links to be more efficient rather than searching through whole network for each selection - refactored how to search for shapes that should be deleted when links are * Separate methods reading of links/nodes/shapes * Refactored node and link creation methods * Separate out model_roadway from RoadwayNetwork class * Separate out various utils to make them easier to find and isolate tricky geo dependencies * Add utility function for creating Point from xy * Added EXPLICIT_LINK_IDENTIFIERS constant as separate from unique_model_link_identifiers so that we can use identifiers such as osm_link_id which may not be unique because of link splitting, but are sufficient for things like specifying links for project cards. Improved: - added validation of new links and nodes - Make logger error reporting slightly more elegant. - made tests quicker by using small example network instead of large network - edited tests to be flexible to network tested - Made networks and directories test fixtures for simplicity - Updated roadway change tests to actually test if change is accurate. - turn print statements to logging - To better manage and run them, tests for different apply methods are separated in different files Fixed: - Fixed types not coercing to strings to do search - corrected occurrences of in/outboundReferenceId --> in/outboundReferenceId*s* - added missing node from small ex network - indexing when created new links and nodes - merge that was creating excess records b/c it should have been an inner join - Lots of debugging on the roadway network and managed lanes side. - Querying if "set" or "existing" or "change" is a key, not if the value of it produces a True b/c a value of 0 is a valid "set"...but produces a False. 📓 Lots of bugs remain in transit, scenario etc to be addressed in separate PRs. PHEW
Moved documentation from `sphinx` to `mkdocs` and set up github action to build and deploy it using `mike` Also: - updated contributing directions - added contributor list Fixes #316 Co-authored-by: Lisa Zorn <lzorn@bayareametro.gov>
Update logging per MTC request 1. separate INFO and DEBUG logs 2. don't auto-call setup_logging() 3. DO auto-call setup_logging() when running tests (added a fixture), putting output in tests/out in filenames which will get overwritten so we don't amass gigs of log files. Also: - fixed some print statements to be logging instead in some tests
also: - update codeblock to be mkdocs instead of sphinx - update .gitignore to ignore built mkdocs site - update specification of watch directory to comply with new mkdocs config instead of mkdocstrings
Fixes #392 Strips non-standardized data from stpaul roadway network per clean_networks.ipynb
- removes notebook output
- Accidentally deleted 'ref' from st paul roadway links b/c wasn't in data model even though it is used. - Added to data model since it is a key OSM field as optional and updated example network to include it. - Updated error checking for the dict query to make sure it is explicitly telling you the field doesn't exist in the df you are querying
Also: - try to fix benchmarking/coverage reports #394 - Fix bug in build osm roadnet.py
Fixes #395 and tests it as well
Also: - link to notebooks that have additional details/info - add info about using included conda env and installing optional dependencies - change name of quickstart to intro since it is actually fairly in depth
- include docs build as part of release and push to avoid a pull clash - update checkout/python action versions for 4 and 5 - separate prerelease and release workflows - prerelease workflow pushes on any main push and wont collide with versions - fix typo referring to other repo - make pyproject.toml dependencies recursive so tests also includes docs and viz
- using coverage.xml
Refactors, bug fixes, performance enhancement, and features for v1 of network_wrangler documented in `notebook/releaseoverviewv1.ipynb` including:
## Features
1. I/O and Mutation Speed
2. Implicit and fast validation
3. Flexible serialization formats
4. Improved stability
5. Selection flexibility for transit based on any feature or by link or node
6. More flexible and fast transition to geodataframes for (almost) any data part
7. Easy data clipping to geographic bounds
8. Scripts for actions you might want to execute from command line (i.e. data conversion/clipping/etc)
9. Logging + Error descriptions
10. Additional project card types: add/delete transit
11. Scripts to create example data from OSM...anywhere
## Tech Overhead Investment
1. Separation of causes prevents circular reference collisions, limits import bloat, and improves legibility/organization.
- Project card functionality in project card repo
- Separate modules for separate functionality
2. Reliability achieved through more testable code and expanding test coverage with more test cases and also anti-patterns
- Testable code
- Test coverage
3. Explicit data models that make complex data structures obvious and easy to validate
- Legible
- Self-documenting
- Flexible
- Validatable
4. Clean code principles that make code easier to test and maintain
- more functions that do a single thing...and do it well
- classes that are small - functions that access or manipulate them
5. Documentation
- consistent and detailed functional documentation
- consistent type hints
- usage for modules and classes
- overall documentation leverages the flexible and less bloated MkDocs package
6. Removes less-well-maintained dependencies
- Replaces Partridge with internal functionality
### Speed
1. Makes the heavy shapes.geojson optional and doesn't read it in for operations that don't directly involve it.
2. Leverages `pandera` for speedy, vector-based data model validation for dataframes
3. Replaces row-based calculations for blank geographic values with vector-based calculations
4. Provides flexibility for I/O serialization formats that are faster – like Parquet.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.