2025.08.19

[atambay37]

Meeting notes:

• Data Sources Integration and Validation: Anshul, Tyler, Peter, and Andrea discussed integrating three main data sources—Land IQ (polygon geospatial), NASS (tabular), and feedstock composition data (tabular)—with a focus on using Land IQ as the foundational geospatial layer and ensuring robust crosswalks and data validation between sources.
  • Data Source Formats: The team confirmed that Land IQ provides polygon geospatial data, while NASS and feedstock composition data are tabular and require merging onto geospatial shapefiles. Land IQ will serve as the base layer for integrating attribute data from the other sources.
  • Crosswalks and Data Validation: Anshul and Peter emphasized the need for a defined system to standardize crop types across datasets, addressing issues where crops may be grouped differently or named inconsistently. The lowest common denominator for crop classification will be Land IQ, and the team will implement data validation to prevent misassignment of attributes.
  • Geographical Scope for MVP: To simplify the initial implementation, the team agreed to focus only on crops present in the North San Joaquin River Valley, reducing the complexity of crosswalks and data requirements for the MVP.
  • Temporal Data Considerations: Peter raised the possibility of incorporating temporal data such as harvest and processing dates to track residue availability over time. The group decided this feature is out of scope for the MVP due to current data limitations but should be considered for future development.
• Vector Versus Raster Data Decision: The group, led by Anshul and Tyler, decided to use vector data as the primary geospatial format for the California tool, citing maintainability, developer community support, and alignment with the existing national biociting tool, while also discussing future support for raster data conversion.
  • Rationale for Vector Format: Tyler explained that vector data is better supported by Geo Django and PostGIS, is more maintainable, and aligns with the tools used in the national biociting tool. The team agreed that vector data is more suitable for their use case, which involves discrete features rather than continuous surfaces.
  • Handling Raster Data: The team discussed the possibility of future data sources arriving in raster format. Tyler described a precedent for extracting pixel values from rasters and converting them to vector points, which can then be aggregated as needed. The group agreed to consider developing utilities for raster-to-vector conversion if required.
  • Granularity and Storage Considerations: The team clarified that the lowest level of granularity in Land IQ is the field, not the plot, which mitigates concerns about vector data storage size. There are no known public datasets at the plot level, so field-level granularity is sufficient.
  • Developer Community Practices: Tyler and Peter noted that vector data is more widely adopted for web-based spatial analysis, especially with Django/Geo Django, and is easier to query and maintain. Raster data would require more custom development and is less suited to their current needs.
• Product Requirements and User Stories Alignment: Anshul led a review of the product requirements document and user stories, confirming with Tyler and Peter that most UI features exist at the national level, but new compositional data in California will require additional features and filters.
  • UI Feature Parity and Gaps: Anshul confirmed with Tyler that the UI tools needed for the California tool are largely present in the national tool, but Peter pointed out that new compositional data (e.g., nitrogen content) will require additional filters and features not present in the national tool.
  • Separation of Frontend and Backend Responsibilities: The team agreed that Don and Niki will focus on backend data preparation and pipeline development, while Tyler will handle frontend feature development, especially those related to new compositional data.
  • Export and API Requirements: Anshul and Tyler discussed the need for a REST API to support data export in multiple formats (e.g., CSV, GeoJSON, shapefile), noting that the national tool currently only supports CSV export. The California tool will require a dedicated API, potentially using Geo Django or FastAPI.
• Technical Stack and Deployment Decisions: The team, including Anshul, Tyler, and Peter, discussed and compared technical stack options for the backend API (Geo Django vs. FastAPI), database hosting (Supabase vs. Google Cloud), and deployment strategies, aiming for flexibility, scalability, and ease of collaboration.
  • Backend Framework Selection: Anshul proposed using FastAPI with SQLModel and SQLAlchemy for the backend API, citing its modularity, serverless deployment options, and strong data validation with Pydantic. Tyler expressed openness to this approach, given the team's experience, but noted that Geo Django is also a mature option.
  • Database Hosting Strategy: Peter and Anshul discussed starting with Supabase for collaborative development, with the option to migrate to Google Cloud if needed for cost or integration reasons. The database will be in Postgres, ensuring portability between platforms.
  • Frontend and Data Pipeline: The California tool will use Next.js for the frontend and Mapbox for vector tile hosting. Data processing, especially for compositional and NASS data, will be handled in R before being loaded into Postgres.
  • Containerized Deployment and Cloud Integration: The team agreed on containerized deployment for portability and scalability, with the frontend likely hosted on Google Cloud Run and the backend API potentially on Google Cloud or another platform, depending on access and collaboration needs.
• Vector Tile Generation and Mapbox Integration: Anshul and Tyler clarified the process for generating and serving vector tiles, confirming that tiles will be pre-generated from Postgres data and hosted on Mapbox for fast rendering, with API endpoints providing detailed data on user interaction.
  • Tile Generation Workflow: The team outlined that after ETL processing and loading data into Postgres, vector tiles will be generated and uploaded to Mapbox. This enables fast map rendering, while detailed attribute queries will be handled via API endpoints.
  • Client-Side Versus Backend Queries: Tyler explained that simple spatial queries can be handled client-side using Turf.js, while more complex queries will be processed by the backend API, following a hybrid approach similar to the national tool.
• User Testing and Stakeholder Engagement: Anshul, Peter, and Tyler discussed plans for user testing, identifying Beam Circular as a key API tester but noting potential delays, and agreed to seek additional users from the lab and community for both API and frontend feedback.
  • Beam Circular's Role: Beam Circular will primarily test the API endpoint rather than the frontend, as their own portal development is on a separate timeline. The team acknowledged that Beam's API testing may be delayed due to their development schedule.
  • Alternative User Testing Plans: The group agreed to identify other users within the lab and broader community to participate in user testing for both the API and the frontend, following an iterative feedback process similar to open source development practices.

Follow-up tasks:

• Data Crosswalk Validation: Develop and solidify the crosswalks between Land IQ, NASS, and feedstock composition data to create a finalized dataset for the MVP. (Peter, Tyler, Anshul)
• Compositional Data Integration: Add new front-end features to the California tool to support compositional data (e.g., filtering by nitrogen content) once the back-end pipeline is ready. (Tyler)
• API Export Options: Implement multimodal export options (e.g., GeoJSON, CSV, shapefile) for the California tool's API to support different user needs. (Tyler)
• API Hosting Access: Investigate and arrange Google Cloud access and permissions for external collaborators to host the API services. (Tyler)
• User Testing Planning: Identify and recruit representative users for both API and front-end user testing, considering the risk that Beam's API testing may be delayed due to their development timeline. (Anshul)

[atambay37]

Hi Team, as you can see the follow-up tasks from this meeting are a lot more about the project at large rather than anything we need to accomplish before our next meeting. I will be working on fleshing out the remaining portions of the PRD and reflecting our discussion. Then we can review, set timelines, and begin submitting PRs!