feat: User Scoping & Artifact Persistence (ADR-082/083) #196
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Migration 034: User Scoping Groups (ADR-082) - kg_auth.groups table with system groups (public, admins) - kg_auth.user_groups for group membership - kg_auth.resource_grants for instance-level access - System user (ID 1) for system-owned resources - Sequences reset to 1000 for regular users/groups - has_access() function for grant-based permission checks Migration 035: Artifact Persistence (ADR-083) - kg_api.query_definitions for saved query recipes - kg_api.artifacts for computed result metadata - Graph epoch freshness tracking via get_graph_epoch() - is_artifact_fresh() helper function - Support for inline results (<10KB) or Garage pointers
…082) - Migration 020: Create system user (ID 1) and admin user (ID 1000) - Migration 034: Add admin user to admins group, document ID conventions - Operator: Add new admin users to admins group on creation - API: Add admin-role users to admins group via POST /users ID ranges follow Unix convention: - 1-999: System/reserved (system user, public/admins groups) - 1000+: Regular users and user-defined groups
…istence (ADR-083)
- Create ArtifactStorageService with size-based routing
- Small payloads (<10KB): stored inline in PostgreSQL
- Large payloads (>=10KB): stored in Garage S3
- Key format: artifacts/{artifact_type}/{artifact_id}.json
- Add prepare_for_storage() for automatic routing decision
- Add factory function get_artifact_storage() for singleton access
- Export service and INLINE_THRESHOLD_BYTES constant
… 3a)
Implement CRUD API endpoints for artifact persistence:
- GET /artifacts - List artifacts with filtering (type, representation, ontology)
- GET /artifacts/{id} - Get artifact metadata
- GET /artifacts/{id}/payload - Get artifact with full payload
- POST /artifacts - Create artifact with automatic storage routing
- DELETE /artifacts/{id} - Delete artifact and cleanup Garage storage
Add CLI commands for validation:
- kg artifact list - List user's artifacts
- kg artifact show <id> - Show artifact metadata
- kg artifact payload <id> - Get full payload
- kg artifact create -t <type> - Create test artifact
- kg artifact delete <id> - Delete artifact
Storage routing automatically selects inline (<10KB) or Garage (>=10KB).
Ownership checks enforce user can only see/modify their own artifacts.
Graph epoch tracking enables freshness detection for stale artifacts.
Implement group management and resource grants:
- GET /groups - List groups with member counts
- POST /groups - Create group (admin only, IDs 1000+)
- GET /groups/{id}/members - List group members
- POST /groups/{id}/members - Add member to group
- DELETE /groups/{id}/members/{user_id} - Remove member
Resource grant endpoints:
- POST /grants - Create resource access grant
- GET /resources/{type}/{id}/grants - List grants for resource
- DELETE /grants/{id} - Revoke grant
Add CLI commands for validation:
- kg group list/members/create/add-member/remove-member
All endpoints require OAuth authentication.
Admin-only operations enforce role check.
Implement query definitions CRUD for saved query recipes:
- GET /query-definitions - List with type filtering
- GET /query-definitions/{id} - Get by ID
- POST /query-definitions - Create new definition
- PUT /query-definitions/{id} - Update definition
- DELETE /query-definitions/{id} - Delete definition
Definition types: block_diagram, cypher, search, polarity, connection
Add CLI commands for validation:
- kg query-def list/show/create/delete
Artifact regeneration endpoint deferred to Phase 4 (job integration).
Major fixes: - Add verify_resource_ownership() helper for grant authorization - Clarify system resource policy: NULL owner_id = system-owned (ID 1) - Resource owners can now create/revoke grants on their resources Minor fixes: - Add error logging for Garage deletion failures in artifact delete - Add composite index (owner_id, representation) for artifact queries - Document double JSON serialization trade-off in artifact_storage - Add include_implicit param to public group members endpoint - Add implicit_membership field to GroupMemberList response Refs: ADR-082, ADR-083
Job Queue Changes: - Migration 036: Add artifact_id column to jobs table - Update job_queue.update_job() to support artifact_id updates Worker Artifact Helper: - Create artifact_helper.py with create_job_artifact() function - Encapsulates artifact creation pattern for workers - Handles graph epoch, storage routing, and job linking Projection Worker Integration: - Add create_artifact flag to job_data - Create artifact on completion when flag is set - Link artifact to job via artifact_id column - Include artifact_id in job result This establishes the pattern for other workers to create artifacts. Polarity analysis and cleanup workers deferred to future work. Refs: ADR-083
Add polarity analysis async job support:
- Create polarity_worker.py with artifact creation
- Add /polarity-axis/jobs endpoint for async analysis
- Update CLI kg polarity analyze with --save-artifact flag
- Add submitPolarityJob client method
Add artifact regeneration:
- Add POST /artifacts/{id}/regenerate endpoint
- Support polarity_analysis and projection types
- Trigger async job using stored parameters
Add artifact cleanup scheduled job:
- Create artifact_cleanup_worker.py for expired artifacts
- Create ArtifactCleanupLauncher for daily 2 AM schedule
- Register worker and scheduled job in migration 036
Also:
- Add BaseModel import fixes in queries.py and artifacts.py
- Add artifact_id to JobResult TypeScript type
- Update tracking file with completed items
- Add `kg search <term>` shortcut (simplifies `kg search query <term>`) - Add `kg help commandmap` for introspective CLI structure tree - Add Unix shortcuts: `kg ls artifact/source`, `kg cat artifact/source`, `kg rm artifact` - Add `kg source list` command with ontology filter - Add `/sources` API endpoint for listing source nodes - Align descriptions in commandmap output for readability - Use terminal width for description truncation (not fixed 45 chars) - Add visual spacing between command groups in commandmap - Document storage architecture (STORAGE-ARCHITECTURE.md)
Split the 1593-line vocabulary.ts into 9 focused modules: - status.ts: status, list commands (~270 lines) - consolidate.ts: consolidate, merge commands (~165 lines) - embeddings.ts: generate-embeddings, category-scores, refresh-categories (~190 lines) - similarity.ts: similar, opposite, analyze commands (~190 lines) - config.ts: config command (~180 lines) - profiles.ts: profiles list/show/create/delete (nested subcommands, ~170 lines) - epistemic.ts: epistemic-status list/show/measure (~330 lines) - sync.ts: sync command (~85 lines) - index.ts: main command wiring (~50 lines) Key improvements: - All files under 350 lines (target: <500) - profiles-* commands now use proper nested subcommands - Removed deprecated config-update command - Better separation of concerns
- Renamed `search details` to `search show` for consistency - Added `details` as alias for backwards compatibility - Aligns with pattern: system resources use `info`, user entities use `show`
- admin.ts (1124 lines) → admin/ directory (5 files): - utils.ts (391 lines): prompt, promptPassword, promptHoldEnter, job tracking - status.ts (106 lines): system status command - scheduler.ts (112 lines): scheduler status/cleanup commands - backup.ts (381 lines): backup, list-backups, restore commands - index.ts (62 lines): command wiring - ai-config.ts (1036 lines) → ai-config/ directory (5 files): - embedding.ts (638 lines): all embedding configuration commands - extraction.ts (181 lines): extraction config/set commands - keys.ts (172 lines): API keys list/set/delete commands - utils.ts (51 lines): shared prompt utilities - index.ts (21 lines): re-exports ADR-082/083 Phase 5 CLI improvements
…-083 Phase 6) - Add --save-artifact flag to kg projection regenerate - Add --save-artifact flag to kg search query and direct search shortcut - Add create_artifact() helper for sync operations (no job linkage) - Update projection API to support artifact creation parameter - Improve install.sh to suppress verbose build output (show on failure only) Artifacts are persisted with type 'projection' or 'search_result' and representation 'cli', enabling recall and sharing of analysis results.
…ase 7) - Add artifact tool with list, show, and payload actions - Update source tool to handle both text and image sources - Text sources return full_text with metadata (document, paragraph, offsets) - Image sources return base64 image as before - Update Phase 6 tracking in todo file MCP server now provides complete artifact management for AI agents to reuse stored analyses without re-running expensive computations.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Implements ADR-082 (User Scoping & Artifact Ownership) and ADR-083 (Artifact Persistence Pattern), completing Phases 1-7.
ADR-082: User Scoping
public(implicit membership) andadminsgroupsADR-083: Artifact Persistence
search_result,projection,polarity_analysis,query_resultCLI Enhancements (Phase 5-6)
kg artifact list/show/payload/deletecommandskg source list/info/getcommandskg group list/members/create/add-member/remove-membercommandskg query-def list/show/create/deletecommandskg ls,kg cat,kg rm,kg statkg search <term>direct shortcutkg help commandmapfor introspective command tree--save-artifactflag forkg projection regenerateandkg search queryMCP Tools (Phase 7)
artifacttool with list/show/payload actionssourcetool handles both text and image sourcesChanges by Phase
Phase 1: Schema Foundation
Phase 2: Artifact Schema
Phase 3a-c: APIs
Phase 4: Async Job Integration
Phase 5: CLI Consolidation
Phase 6: CLI Enhancements
--save-artifactflags for projection and searchPhase 7: MCP Tools
Test plan
npm run buildin cli/kg artifact listshows artifactskg search "term" --save-artifactcreates artifactkg projection regenerate <ont> --save-artifact --forcecreates artifactkg artifact show <id>displays metadatakg group listshows public and admins groups