perf: pre-fetch CSO external ID dictionary for O(1) import matching

.claude/skills/release/SKILL.md

@@ -2,7 +2,6 @@
 name: release
 description: Create a new JIM release — updates VERSION, CHANGELOG, PowerShell manifest, commits, tags, and pushes
 argument-hint: "<version> (e.g., 0.4.0 or 0.4.0-alpha)"
-allowed-tools: Bash, Read, Edit, Write, Glob, Grep
 ---
 
 # Create a JIM Release
@@ -39,6 +38,121 @@ Before starting, verify:
 
 5. **Review pinned Docker dependencies**: Check that base image digests and apt package versions are current. If Dependabot PRs for Docker digests have been merged since the last release, flag this so the user can verify pinned apt versions still match.
 
+## Documentation Review and Update
+
+Before validating the changelog, ensure all documentation reflects the current state of the codebase.
+
+### 1. Identify what changed since the last release
+
+Get the last release tag and list all commits since then:
+```
+git describe --tags --abbrev=0
+git log <last-tag>..HEAD --oneline --no-merges
+```
+
+From this list, identify commits that introduced:
+- New features or capabilities
+- Changed behaviour or configuration
+- New or modified API endpoints
+- New connectors, components, or services
+- Architectural changes (new projects, containers, dependencies)
+- Changed deployment or setup steps
+
+### 2. Review and update documentation
+
+Cross-reference the identified changes against ALL of the following documentation files, and update any that are out of date:
+
+- **`README.md`** (root) — Feature list, quick-start instructions, architecture overview, prerequisites
+- **`docs/DEVELOPER_GUIDE.md`** — Architecture guide, development workflows, component documentation, diagram references
+- **`docs/DEPLOYMENT_GUIDE.md`** — Deployment instructions, configuration, environment variables
+- **`docs/DATABASE_GUIDE.md`** — Schema changes, migration notes, database configuration
+- **`docs/EXPRESSIONS_GUIDE.md`** — Expression language documentation, available functions
+- **`docs/TESTING_STRATEGY.md`** — Testing approach, test categories, integration test documentation
+- **`docs/COMPLIANCE_MAPPING.md`** — Security and compliance documentation
+- **`docs/CACHING_STRATEGY.md`** — Caching architecture and configuration
+- **`docs/RELEASE_PROCESS.md`** — Release steps (if process has changed)
+- **`docs/JIM_AI_ASSISTANT_CONTEXT.md`** — AI assistant context document
+- **`docs/JIM_AI_ASSISTANT_INSTRUCTIONS.md`** — AI assistant instructions
+- **Any other `.md` files in `docs/`** that are affected by the changes
+
+For each file, check whether the changes since the last release have made any content inaccurate, incomplete, or missing. Make the updates directly — do not just flag them.
+
+### 3. Review and update architectural diagrams
+
+#### C4 Model (Structurizr)
+
+1. **Review `docs/diagrams/structurizr/workspace.dsl`** against the current codebase. Check that:
+   - All containers (services, databases) are represented
+   - All components within each container are current
+   - Relationships between components/containers are accurate
+   - Any new connectors, services, or significant components added since the last release are included
+   - Removed or renamed components are cleaned up
+
+2. **Update the DSL** if any changes are needed.
+
+3. **Regenerate the C4 SVG images**:
+   ```
+   jim-diagrams
+   ```
+   This exports both light and dark theme SVGs to `docs/diagrams/images/`.
+
+4. **Review `docs/diagrams/structurizr/docs/01-overview.md`** — update the diagram documentation page if the diagrams have changed or new views have been added.
+
+#### Mermaid Process Diagrams
+
+Review each Mermaid diagram in `docs/diagrams/mermaid/` against the current codebase behaviour:
+
+- `FULL_IMPORT_FLOW.md` — Object import, duplicate detection, deletion detection
+- `FULL_SYNC_CSO_PROCESSING.md` — Per-CSO sync decision tree
+- `DELTA_SYNC_FLOW.md` — Delta sync watermark and early exit logic
+- `EXPORT_EXECUTION_FLOW.md` — Export batching, parallelism, retry
+- `PENDING_EXPORT_LIFECYCLE.md` — Pending export state machine
+- `WORKER_TASK_LIFECYCLE.md` — Worker polling, dispatch, heartbeat
+- `SCHEDULE_EXECUTION_LIFECYCLE.md` — Schedule step groups and recovery
+- `CONNECTOR_LIFECYCLE.md` — Connector interface and lifecycle
+- `ACTIVITY_AND_RPEI_FLOW.md` — Activity and RPEI accumulation
+- `MVO_DELETION_AND_GRACE_PERIOD.md` — Deletion rules and grace periods
+
+For each diagram, check whether the logic or flow has changed since the last release. Update any diagrams that no longer accurately reflect the code.
+
+#### Diagram references in documentation
+
+Verify that pages embedding or linking to diagrams are up to date:
+- `README.md` — SVG references
+- `docs/DEVELOPER_GUIDE.md` — C4 and Mermaid diagram listings and links
+- `docs/diagrams/structurizr/README.md` — Structurizr tooling documentation
+
+If new diagrams were added or existing ones renamed/removed, update these references accordingly.
+
+### 4. Review marketing site content
+
+Fetch the public marketing site at `https://tetron.io/jim/` and compare its content against the current state of JIM:
+
+1. **Feature claims** — Are all listed features still accurate? Are significant new features missing?
+2. **Architecture descriptions** — Does the site's description of JIM's architecture match the current state?
+3. **Connector/integration list** — Are all supported connectors and integrations listed?
+4. **Deployment/requirements** — Are prerequisites and deployment descriptions current?
+5. **Screenshots or visuals** — Are they representative of the current UI?
+
+The release agent cannot update the marketing site directly. Instead, present a clear summary of recommended changes to the user, structured so that a website development agent can implement them easily:
+- List each recommended change with: **what to change**, **where on the page**, **what the current text/content says**, and **what it should say or show instead**
+- Include any new feature descriptions that should be added, written in marketing-appropriate language
+- Flag any content that should be removed (e.g., features that were deprecated or renamed)
+
+### 5. Present documentation changes for review
+
+Show the user a summary of all documentation and diagram changes made:
+- List of files updated and what changed in each
+- Any new diagrams added
+- Marketing site recommendations (for manual action)
+- Ask the user to confirm before proceeding to changelog validation
+
+Once confirmed, commit the documentation updates:
+```bash
+git add README.md docs/ docs/diagrams/
+git commit -m "docs: update documentation and diagrams for v<version> release"
+```
+
 ## Changelog Validation
 
 Before proceeding with version updates, validate that the changelog is complete.

.devcontainer/CLAUDE.md

@@ -0,0 +1,165 @@
+# Development Environment Reference
+
+> Commands, workflows, environment setup, and troubleshooting. See root `CLAUDE.md` for behavioural rules and guardrails.
+
+## Commands
+
+**Build & Test:**
+- `dotnet build JIM.sln` - Build entire solution (use for final pre-PR check)
+- `dotnet build test/JIM.Worker.Tests/` - Build a specific project and its dependencies (preferred during development)
+- `dotnet test JIM.sln` - Run all tests (use for final pre-PR check)
+- `dotnet test test/JIM.Worker.Tests/` - Run a specific test project (preferred during development)
+- `dotnet test --filter "FullyQualifiedName~TestName"` - Run specific test
+- `dotnet clean && dotnet build` - Clean build
+
+**Database:**
+- `dotnet ef migrations add [Name] --project src/JIM.PostgresData` - Add migration
+- `dotnet ef database update --project src/JIM.PostgresData` - Apply migrations
+- `docker compose exec jim.web dotnet ef database update` - Apply migrations in Docker
+
+**Shell Aliases (Recommended):**
+- Aliases are automatically configured from `.devcontainer/jim-aliases.sh`
+- If aliases don't work, run: `source ~/.zshrc` (or restart terminal)
+- `jim` - List all available jim aliases
+- `jim-compile` - Build entire solution (dotnet build)
+- `jim-test` - Run unit + workflow tests (excludes Explicit)
+- `jim-test-all` - Run ALL tests (incl. Explicit + Pester)
+- `jim-db` - Start PostgreSQL (for local debugging)
+- `jim-db-stop` - Stop PostgreSQL
+- `jim-migrate` - Apply migrations
+- `jim-postgres-tune` - Re-tune PostgreSQL for current CPU/RAM
+
+**Docker Stack Management:**
+- `jim-stack` - Start Docker stack
+- `jim-stack-logs` - View Docker stack logs
+- `jim-stack-down` - Stop Docker stack
+- `jim-restart` - Restart stack (re-reads .env, no rebuild)
+
+**Docker Builds (rebuild and start services):**
+- `jim-build` - Build all services + start
+- `jim-build-web` - Build jim.web + start
+- `jim-build-worker` - Build jim.worker + start
+- `jim-build-scheduler` - Build jim.scheduler + start
+
+**Reset:**
+- `jim-reset` - Reset JIM (delete database & logs volumes)
+
+**Diagrams:**
+- `jim-diagrams` - Export Structurizr C4 diagrams as SVG (requires Docker)
+
+**Planning:**
+- `jim-prd` - Create a new PRD from template (prompts for feature name)
+
+**Docker (Manual Commands):**
+- `docker compose -f db.yml up -d` - Start database (same as jim-db)
+- `docker compose -f db.yml down` - Stop database
+- `docker compose logs [service]` - View service logs
+
+**IMPORTANT - Rebuilding Containers After Code Changes:**
+When running the Docker stack and you make code changes to JIM.Web, JIM.Worker, or JIM.Scheduler, you MUST rebuild the affected container(s) for changes to take effect:
+- `jim-build-web` - Rebuild and restart jim.web service
+- `jim-build-worker` - Rebuild and restart jim.worker service
+- `jim-build-scheduler` - Rebuild and restart jim.scheduler service
+- `jim-build` - Rebuild and restart all services
+
+Blazor pages, API controllers, and other compiled code require container rebuilds. Simply refreshing the browser will not show changes.
+
+**PostgreSQL Auto-Tuning:**
+PostgreSQL is automatically tuned during devcontainer setup (`.devcontainer/postgres-tune.sh`). The script:
+- Auto-detects CPU cores and available RAM
+- Calculates optimal pgtune settings for OLTP workloads
+- Generates two gitignored overlay files: `docker-compose.local.yml` and `db.local.yml`
+- The `jim-*` aliases automatically include these overlays when present (later files win)
+
+If you later increase devcontainer resources (e.g., scale from 4c/8GB to 8c/32GB), re-tune and restart:
+- `jim-postgres-tune` then `jim-db-stop && jim-db` (local dev)
+- `jim-postgres-tune` then `jim-restart` (Docker stack)
+
+## Dependency Update Policy
+
+All dependency updates from Dependabot require human review before merging - there is no auto-merge. This applies to all ecosystems: NuGet packages, Docker base images, and GitHub Actions. A maintainer must review each PR, verify the changes are appropriate, and merge manually.
+
+**NuGet Packages:**
+- Pin dependency versions in `.csproj` files (avoid floating versions)
+- Dependabot proposes weekly PRs for patch and minor updates
+- Review each update for compatibility, changelog notes, and known issues before merging
+- Run `dotnet list package --vulnerable` to check for known vulnerabilities
+
+**Docker Base Images:**
+- Production Dockerfiles pin base image digests (`@sha256:...`) and functional apt package versions for reproducible builds
+- **NEVER** remove the `@sha256:` digest from `FROM` lines
+- **NEVER** remove version pins from functional apt packages (libldap, cifs-utils)
+- Diagnostic utilities (curl, iputils-ping) are intentionally unpinned
+- If updating a base image digest, check and update pinned apt versions to match (see `docs/DEVELOPER_GUIDE.md` "Dependency Pinning" section)
+
+**GitHub Actions:**
+- Pin action versions by major version tag (e.g., `@v4`) in workflow files
+- Dependabot proposes weekly PRs for patch and minor updates
+- Review each update before merging
+
+## Development Workflows
+
+**Choose one of two workflows:**
+
+**Workflow 1 - Local Debugging (Recommended):**
+1. Start database: `jim-db`
+2. Press F5 in VS Code or run: `jim-web`
+3. Debug with breakpoints and hot reload
+4. Services: Web + API (https://localhost:7000), Swagger at `/api/swagger`
+
+**Workflow 2 - Full Docker Stack:**
+1. Start all services: `jim-stack`
+2. Access containerized services
+3. Services: Web + API (http://localhost:5200), Swagger at `/api/swagger`
+
+**Use Workflow 1** for active development and debugging.
+**Use Workflow 2** for integration testing or production-like environment.
+
+## Environment Setup
+
+**Required:**
+- .NET 9.0 SDK
+- PostgreSQL 18 (via Docker)
+- Docker & Docker Compose
+
+**Configuration:**
+- Copy `.env.example` to `.env`
+- Set database credentials
+- Configure SSO/OIDC settings (required)
+
+**Optional Environment Variables:**
+- `JIM_ENCRYPTION_KEY_PATH` - Custom path for encryption key storage (default: `/data/keys` for Docker, or app data directory)
+
+**GitHub Codespaces:**
+- PostgreSQL memory settings automatically optimized
+- Use `jim-db` or `jim-stack` aliases (already configured)
+
+## Troubleshooting
+
+**Build fails:**
+- Check .NET 9.0 SDK installed: `dotnet --version`
+- Restore packages: `dotnet restore JIM.sln`
+
+**Tests fail:**
+- Verify test method signature: `public async Task TestNameAsync()`
+- Check `Assert.ThrowsAsync` is awaited: `await Assert.ThrowsAsync<Exception>(...)`
+
+**Database connection:**
+- Verify PostgreSQL running: `docker compose ps`
+- Check `.env` connection string
+- Apply migrations if needed
+
+**Sync Activities Failing with UnhandledError:**
+- UnhandledError items in Activities indicate code/logic bugs, not data problems
+- Check worker logs for the full exception stack trace using: `docker compose logs jim.worker --tail=1000 | grep -A 5 "Unhandled.*sync error"`
+- Common causes:
+  - Query returning unexpected number of results (e.g., `SingleOrDefaultAsync` with duplicates)
+  - Missing or null data where code expected values
+  - Type casting errors or invalid data states
+- DO NOT silently ignore UnhandledErrors - they indicate data integrity risk
+
+**Sync Statistics Not What Expected:**
+- Check log for summary statistics at end of import/sync/export (look for "SUMMARY - Total objects")
+- Verify Run Profile is selecting correct partition/container
+- Verify sync rules are correctly scoped to object types
+- Check for DuplicateObject errors - indicates deduplication is working

CHANGELOG.md

@@ -18,6 +18,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - ✨ Keycloak admin console accessible at `http://localhost:8181`
 - 🔒 HTTP OIDC authority support for development (RequireHttpsMetadata conditionally disabled)
 
+### Performance
+
+- ⚡ Import CSO matching now uses a pre-fetched dictionary for O(1) external ID lookups, replacing N per-object database queries with a single bulk query at import start — eliminates the dominant bottleneck in full imports (#440)
+
 ### Fixed
 
 - 🔒 Attribute change history is no longer cascade-deleted when a metaverse or connected system attribute definition is removed — the FK is set to null and snapshot `AttributeName`/`AttributeType` properties preserve the audit trail indefinitely (#58)