Skip to content

feat: add configurable local data directory with stale lock recovery#4

Merged
frperezr merged 1 commit intokraklabs:mainfrom
rodrwan:feat/configurable-data-dir
Feb 14, 2026
Merged

feat: add configurable local data directory with stale lock recovery#4
frperezr merged 1 commit intokraklabs:mainfrom
rodrwan:feat/configurable-data-dir

Conversation

@rodrwan
Copy link
Copy Markdown
Contributor

@rodrwan rodrwan commented Feb 14, 2026

Description

Add support for a configurable local data directory, replacing the hardcoded ~/.cie/data/ path. Users can now control where CIE stores indexed data via the indexing.local_data_dir config field or the CIE_DATA_DIR environment variable. Relative paths are resolved from the directory containing .cie/project.yaml.

Additionally includes stale RocksDB LOCK file recovery (auto-removes orphaned locks from crashed processes), graceful shutdown improvements, and build system fixes.

Related Issue

N/A

Type of Change

  • New feature (non-breaking change that adds functionality)
  • Documentation update
  • CI/CD or build changes

Testing

  • I have tested this locally
  • I have added/updated tests that prove my fix/feature works
  • New and existing tests pass locally with my changes

Tests added

  • cmd/cie/paths_test.go — unit tests for projectDataDir, dataRootFromConfig, path resolution (absolute, relative, default fallback)
  • pkg/storage/embedded_test.goTestIsStaleLock_NoLockFile, TestIsStaleLock_StaleLock, TestIsStaleLock_ActiveLock

Checklist

  • My code follows the project's code style
  • I have run make fmt and make lint
  • I have added comments for complex code sections
  • I have updated documentation if needed
  • My changes generate no new warnings

Key Changes

Configurable data directory

Method Example Behavior
Config field indexing.local_data_dir: "/mnt/cie-data" Absolute path used as-is, /<project_id> appended
Config field (relative) indexing.local_data_dir: ".cie-data" Resolved from project config directory
Env var CIE_DATA_DIR=/mnt/cie-data Overrides config field
Default (unset) ~/.cie/data/<project_id>/ (unchanged behavior)

Legacy data migration warning

When local_data_dir is configured and data exists at the old default path (~/.cie/data/<project_id>/) but not at the new path, CIE prints a warning guiding the user to re-index.

Stale RocksDB LOCK recovery (pkg/storage/embedded.go)

If CozoDB fails to open and a LOCK file exists, CIE checks whether any process actually holds the flock. If not (crashed/killed process), it removes the stale lock and retries. This eliminates the need for users to manually delete lock files after crashes.

Graceful shutdown improvements

  • All signal handlers now call signal.Stop(sigChan) so a second Ctrl+C force-quits immediately
  • User-facing message: "Shutting down... Press Ctrl+C again to force quit."
  • Applied to: index, mcp, and serve commands

Build fixes (Makefile, pkg/cozodb/cozodb.go)

  • INSTALL_DIR uses $(GOPATH)/bin instead of $(HOME)/go/bin
  • macOS: ad-hoc codesign after install (codesign --force --sign -)
  • CGO LDFLAGS: -L${SRCDIR}/../../lib so go install ./cmd/cie finds the vendored static library

Refactoring

  • Centralized path resolution in new cmd/cie/paths.go (replaces scattered filepath.Join(homeDir, ".cie", "data", ...) calls)
  • Moved checkLocalData logic into LocalPipeline.FunctionCount() — queries through the already-open DB instead of opening a second connection
  • storage.EmbeddedConfig.DataDir is now passed explicitly instead of being computed internally

Additional Notes

  • Backward compatible: when local_data_dir is not set, behavior is identical to before
  • The EmbeddedConfig.DataDir field replaces the internal ~/.cie/data/<project> computation in storage.NewEmbeddedBackend, giving callers full control over the path

Allow users to customize where CIE stores indexed data via
`indexing.local_data_dir` config or `CIE_DATA_DIR` env var, instead of
hardcoding `~/.cie/data/`. Relative paths resolve from the project config
directory. Includes legacy data dir detection with migration warning,
stale RocksDB LOCK file recovery, graceful shutdown improvements
(signal.Stop + user message), and build fixes (GOPATH/bin, macOS
codesign, CGO LDFLAGS with ${SRCDIR}).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@frperezr
Copy link
Copy Markdown
Member

frperezr commented Feb 14, 2026

Code review

No issues found. Checked for bugs and CLAUDE.md compliance. Tests pass across all packages.

@frperezr frperezr merged commit 5b74b28 into kraklabs:main Feb 14, 2026
1 of 4 checks passed
frperezr added a commit that referenced this pull request Feb 14, 2026
- Remove unused getCIEDir() function that golangci-lint flagged
- Fix Linux build: copy platform-specific libcozo_c.a to repo lib/
  instead of system paths, so CGO LDFLAGS -L${SRCDIR}/../../lib
  finds the correct library (not the vendored macOS one)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants