feat: Style Guide & Scripts for Extracting Files from Google Drive

AGENTS.md

@@ -404,7 +404,7 @@ ca-biositing/
 │       │   │       │   ├── transform/       # Transform tasks
 │       │   │       │   └── load/            # Load tasks
 │       │   │       ├── flows/               # Prefect flows
-│       │   │       │   ├── primary_product.py
+│       │   │       │   ├── primary_ag_product.py
 │       │   │       │   ├── analysis_type.py
 │       │   │       │   └── ...
 │       │   │       └── utils/               # Utilities

ERD_VIEW.md

@@ -0,0 +1,61 @@
+# ERD for USDA Census and Survey Data
+
+To view this diagram, open the Command Palette (`Cmd+Shift+P` on Mac or
+`Ctrl+Shift+P` on Windows/Linux) and run **"Markdown: Open Preview to the
+Side"**.
+
+```mermaid
+erDiagram
+CensusRecord {
+    integer year
+    CropEnum crop
+    VariableEnum variable
+    UnitEnum unit
+    float value
+    BearingStatusEnum bearing_status
+    string class_desc
+    string domain_desc
+    string source
+    string notes
+}
+Geography {
+    string state_name
+    string state_fips
+    string county_name
+    string county_fips
+    string geoid
+    string region_name
+    string agg_level_desc
+}
+SurveyRecord {
+    string period_desc
+    string freq_desc
+    string program_desc
+    integer year
+    CropEnum crop
+    VariableEnum variable
+    UnitEnum unit
+    float value
+    BearingStatusEnum bearing_status
+    string class_desc
+    string domain_desc
+    string source
+    string notes
+}
+UsdaRecord {
+    integer year
+    CropEnum crop
+    VariableEnum variable
+    UnitEnum unit
+    float value
+    BearingStatusEnum bearing_status
+    string class_desc
+    string domain_desc
+    string source
+    string notes
+}
+
+CensusRecord ||--|o Geography : "geography"
+SurveyRecord ||--|o Geography : "geography"
+UsdaRecord ||--|o Geography : "geography"
+```

README.md

@@ -1,45 +1,299 @@
 # ca-biositing
 
-Discussion of general issues related to the project and protyping or research
+A geospatial bioeconomy project for biositing analysis in California. This
+repository provides tools for ETL pipelines to process data from Google Sheets
+into PostgreSQL databases, geospatial analysis using QGIS, and a REST API for
+data access.
+
+## Project Structure
+
+This project uses a **PEP 420 namespace package** structure with three main
+components:
+
+- **`ca_biositing.datamodels`**: Shared LinkML/SQLModel database models and
+  database configuration
+- **`ca_biositing.pipeline`**: ETL pipelines orchestrated with Prefect, deployed
+  via Docker
+- **`ca_biositing.webservice`**: FastAPI REST API for data access
+
+### Directory Layout
+
+```text
+ca-biositing/
+├── src/ca_biositing/           # Namespace package root
+│   ├── datamodels/             # Database models (SQLModel)
+│   ├── pipeline/               # ETL pipelines (Prefect)
+│   └── webservice/             # REST API (FastAPI)
+├── resources/                  # Deployment resources
+│   ├── docker/                 # Docker Compose configuration
+│   └── prefect/                # Prefect deployment files
+├── tests/                      # Integration tests
+├── pixi.toml                   # Pixi dependencies and tasks
+└── pixi.lock                   # Dependency lock file
+```
+
+## Quick Start
+
+### Prerequisites
+
+- **Pixi** (v0.55.0+):
+  [Installation Guide](https://pixi.sh/latest/#installation)
+- **Docker**: For running the ETL pipeline
+- **Google Cloud credentials**: For Google Sheets access (optional)
+
+### Installation
+
+```bash
+# Clone the repository
+git clone https://github.com/uw-ssec/ca-biositing.git
+cd ca-biositing
+
+# Install dependencies with Pixi
+pixi install
+
+# Install pre-commit hooks
+pixi run pre-commit-install
+```
 
-## Relevant Links for project documentations and context
+### Running Components
 
-- eScience Slack channel: 🔒
-  [#ssec-ca-biositing](https://escience-institute.slack.com/archives/C098GJCTTFE)
-- SSEC Sharepoint (**INTERNAL SSEC ONLY**): 🔒
-  [Projects/GeospatialBioeconomy](https://uwnetid.sharepoint.com/:f:/r/sites/og_ssec_escience/Shared%20Documents/Projects/GeospatialBioeconomy?csf=1&web=1&e=VBUGQG)
-- Shared Sharepoint Directory: 🔒
-  [SSEC CA Biositing Shared Folder](https://uwnetid.sharepoint.com/:f:/r/sites/og_ssec_escience/Shared%20Documents/Projects/GeospatialBioeconomy/SSEC%20CA%20Biositing%20Shared%20Folder?csf=1&web=1&e=p5wBel)
+#### ETL Pipeline (Prefect + Docker)
 
-## General Discussions
+**Note**: Before starting the services for the first time, create the required
+environment file from the template:
 
-For general discussion, ideas, and resources please use the
-[GitHub Discussions](https://github.com/uw-ssec/ca-biositing/discussions).
-However, if there's an internal discussion that need to happen, please use the
-slack channel provided.
+```bash
+cp resources/docker/.env.example resources/docker/.env
+```
 
-- Meeting Notes in GitHub:
-  [discussions/meetings](https://github.com/uw-ssec/ca-biositing/discussions/categories/meetings)
+Then start and use the services:
+
+```bash
+# 1. Create the initial database migration script
+# (This is only needed once for a new database)
+pixi run initial-migration
+
+# 2. Start all services (PostgreSQL, Prefect server, worker)
+# This will also automatically apply any pending database migrations.
+pixi run start-services
+
+# 3. Deploy flows to Prefect
+pixi run deploy
+
+# Run the ETL pipeline
+pixi run run-etl
+
+# Monitor via Prefect UI: http://localhost:4200
+
+# To apply new migrations after the initial setup
+pixi run migrate
+
+# Stop services
+pixi run teardown-services
+```
+
+See [`resources/README.md`](resources/README.md) for detailed pipeline
+documentation.
 
-## Questions
+#### Web Service (FastAPI)
 
-If you have any questions about our process, or locations of SSEC resources,
-please ask [Anshul Tambay](https://github.com/atambay37).
+```bash
+# Start the web service
+pixi run start-webservice
 
-## QGIS
+# Access API docs: http://localhost:8000/docs
+```
 
-This project includes QGIS for geospatial analysis and visualization. You can
-run QGIS using pixi with the following command:
+#### QGIS (Geospatial Analysis)
 
 ```bash
 pixi run qgis
 ```
 
-This will launch QGIS in the `gis` environment with all necessary dependencies
-installed.
+**Note**: On macOS, you may see a Python faulthandler error - this is expected
+and can be ignored. See
+[QGIS Issue #52987](https://github.com/qgis/QGIS/issues/52987).
+
+## Development
+
+### Running Tests
+
+```bash
+# Run all tests
+pixi run test
+
+# Run tests with coverage
+pixi run test-cov
+```
+
+### Code Quality
+
+```bash
+# Run pre-commit checks on staged files
+pixi run pre-commit
+
+# Run pre-commit on all files (before PR)
+pixi run pre-commit-all
+```
+
+### Available Pixi Tasks
+
+View all available tasks:
+
+```bash
+pixi task list
+```
+
+Key tasks:
+
+- **Service Management**: `start-services`, `teardown-services`,
+  `service-status`
+- **ETL Operations**: `deploy`, `run-etl`
+- **Development**: `test`, `test-cov`, `pre-commit`, `pre-commit-all`
+- **Applications**: `start-webservice`, `qgis`
+- **Database**: `access-db`, `check-db-health`
+- **Datamodels**: `update-schema`, `migrate`
+
+## Architecture
+
+### Namespace Packages
+
+This project uses **PEP 420 namespace packages** to organize code into
+independently installable components that share a common namespace:
+
+- Each component has its own `pyproject.toml` and can be installed separately
+- Shared models in `datamodels` are used by both `pipeline` and `webservice`
+- Clear separation of concerns while maintaining type consistency
+
+### ETL Pipeline
+
+The ETL pipeline uses:
+
+- **Prefect**: Workflow orchestration and monitoring
+- **Docker**: Containerized execution environment
+- **PostgreSQL**: Data persistence
+- **Google Sheets API**: Primary data source
+
+Pipeline architecture:
+
+1. **Extract**: Pull data from Google Sheets
+2. **Transform**: Clean and normalize data with pandas
+3. **Load**: Insert/update records in PostgreSQL via SQLModel
+
+### Database Models
+
+We use a **LinkML-first approach** for defining our data schema. The workflow
+is:
+
+1.  **LinkML Schema**: The schema is defined in YAML files (source of truth).
+2.  **SQLAlchemy Generation**: Python classes are automatically generated from
+    LinkML.
+3.  **Alembic Migrations**: Database migrations are generated from the Python
+    classes.
+
+SQLModel-based models provide:
+
+- Type-safe database operations
+- Automatic schema generation (via Alembic)
+- Shared models across ETL and API components
+- Pydantic validation
+
+## Project Components
+
+### 1. Data Models (`ca_biositing.datamodels`)
+
+Database models for:
+
+- Biomass data (field samples, measurements)
+- Geographic locations
+- Experiments and analysis
+- Metadata and samples
+- Organizations and contacts
+
+**Documentation**:
+[`src/ca_biositing/datamodels/README.md`](src/ca_biositing/datamodels/README.md)
+
+### 2. ETL Pipeline (`ca_biositing.pipeline`)
+
+Prefect-orchestrated workflows for:
+
+- Data extraction from Google Sheets
+- Data transformation and validation
+- Database loading and updates
+- Lookup table management
+
+**Documentation**:
+[`src/ca_biositing/pipeline/README.md`](src/ca_biositing/pipeline/README.md)
+
+**Guides**:
+
+- [Docker Workflow](src/ca_biositing/pipeline/docs/DOCKER_WORKFLOW.md)
+- [Prefect Workflow](src/ca_biositing/pipeline/docs/PREFECT_WORKFLOW.md)
+- [ETL Development](src/ca_biositing/pipeline/docs/ETL_WORKFLOW.md)
+- [Database Migrations](src/ca_biositing/pipeline/docs/ALEMBIC_WORKFLOW.md)
+
+### 3. Web Service (`ca_biositing.webservice`)
+
+FastAPI REST API providing:
+
+- Read access to database records
+- Interactive API documentation (Swagger/OpenAPI)
+- Type-safe endpoints using Pydantic
+
+**Documentation**:
+[`src/ca_biositing/webservice/README.md`](src/ca_biositing/webservice/README.md)
+
+### 4. Deployment Resources (`resources/`)
+
+Docker and Prefect configuration for:
+
+- Service orchestration (Docker Compose)
+- Prefect deployments
+- Database initialization
+
+**Documentation**: [`resources/README.md`](resources/README.md)
+
+## Adding Dependencies
+
+### For Local Development (Pixi)
+
+```bash
+# Add conda package to default environment
+pixi add <package-name>
+
+# Add PyPI package to default environment
+pixi add --pypi <package-name>
+
+# Add to specific feature (e.g., pipeline)
+pixi add --feature pipeline --pypi <package-name>
+```
+
+### For ETL Pipeline (Docker)
+
+The pipeline dependencies are managed by Pixi's `etl` environment feature in
+`pixi.toml`. When you add dependencies and rebuild Docker images, they are
+automatically included:
+
+```bash
+# Add dependency to pipeline feature
+pixi add --feature pipeline --pypi <package-name>
+
+# Rebuild Docker images
+pixi run rebuild-services
+
+# Restart services
+pixi run start-services
+```
+
+## Environment Management
+
+This project uses **Pixi environments** for different workflows:
 
-For MacOS, there will be a Python error about faulthandler, which is expected
-and can be ignored, see https://github.com/qgis/QGIS/issues/52987.
+- **`default`**: General development, testing, pre-commit hooks
+- **`gis`**: QGIS and geospatial analysis tools
+- **`etl`**: ETL pipeline (used in Docker containers)
+- **`webservice`**: FastAPI web service
+- **`frontend`**: Node.js/npm for frontend development
 
 ## Frontend Integration