Support configurable embedding vector dimension for OpenAI-compatible providers

[JumpLink]

We use an OpenAI-compatible embeddings API (Scaleway) with the model bge-multilingual-gemma2, which outputs 3584-dimensional vectors. The current schema fixes the vector table to 1536 dimensions in migrations, which causes a dimension mismatch and prevents using this (and other non-1536) models.

This PR makes the vector dimension configurable:

• Migration 012: Drops documents_vec so it is no longer created with a fixed dimension in migrations.
• Runtime: After migrations, DocumentStore.ensureVectorTable() creates documents_vec using config.embeddings.vectorDimension (already exposed via DOCS_MCP_EMBEDDINGS_VECTOR_DIMENSION) and runs the same backfill as in migration 011. Existing triggers continue to apply.

Users can set DOCS_MCP_EMBEDDINGS_VECTOR_DIMENSION=3584 (or the dimension of their model) when using Scaleway or any other provider whose embedding dimension is not 1536. The default remains 1536 for existing setups.

[JumpLink]

I am currently testing the changes in this PR with our stack at faktenforum/ai-chat-interface#17 and it seems to be working well.

[Copilot]

Pull request overview

This PR enables using OpenAI-compatible embedding providers that return non-1536 dimensional vectors by making the SQLite documents_vec virtual table’s embedding dimension configurable at runtime via config.embeddings.vectorDimension.

Changes:

• Add validation to require embeddings.vectorDimension >= 1.
• Add migration 012 to drop documents_vec so it can be recreated at runtime instead of being fixed at 1536 in migrations.
• Add DocumentStore.ensureVectorTable() to create/reconcile documents_vec using the configured dimension and backfill from documents.embedding; update migration tests accordingly.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
src/utils/config.ts	Adds a minimum constraint for embeddings.vectorDimension.
src/store/applyMigrations.test.ts	Updates migration expectations and creates a test-only vec table for vector-search tests.
src/store/DocumentStore.ts	Creates/reconciles documents_vec at startup using configurable dimension and performs backfill.
db/migrations/012-drop-documents-vec-for-runtime.sql	Drops documents_vec during migrations to defer creation to runtime.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

ensureVectorTable() backfills documents_vec directly from documents.embedding without ensuring the JSON array length matches the configured dimension. If a user changes embeddings.vectorDimension (e.g., from 1536 -> 3584), existing stored embeddings will typically still be 1536-long and inserting them into FLOAT[3584] is likely to fail, preventing startup. Consider filtering the backfill to only rows whose embedding length matches dim, or explicitly padding/normalizing (or clearing) existing embeddings when recreating the table so initialization can succeed after a dimension change.

Suggested change

	WHERE d.embedding IS NOT NULL;
	WHERE d.embedding IS NOT NULL
	AND json_array_length(d.embedding) = ${dim};

[Copilot]

ensureVectorTable() uses a check-then-create pattern (SELECT sqlite_master then CREATE VIRTUAL TABLE) without IF NOT EXISTS / DROP ... IF EXISTS or a transaction. If two processes/instances initialize against the same DB concurrently, one can race and throw "table already exists" (or drop conflicts). Making the DDL idempotent (e.g., CREATE VIRTUAL TABLE IF NOT EXISTS, DROP TABLE IF EXISTS) and/or wrapping the reconcile step in a transaction would make startup more reliable under concurrent access.

Suggested change

	const existingSql = this.db
	.prepare(
	"SELECT sql FROM sqlite_master WHERE type = 'table' AND name = 'documents_vec';",
	)
	.get() as { sql: string } | undefined;
	if (existingSql) {
	const match = existingSql.sql.match(/embedding\s+FLOAT\s*\[\s*(\d+)\s*]/i);
	const existingDim = match ? Number(match[1]) : null;
	if (existingDim === dim) {
	return;
	}
	this.db.exec("DROP TABLE documents_vec;");
	}
	this.db.exec(`
	CREATE VIRTUAL TABLE documents_vec USING vec0(
	library_id INTEGER NOT NULL,
	version_id INTEGER NOT NULL,
	embedding FLOAT[${dim}]
	);
	`);
	this.db.exec(`
	INSERT OR REPLACE INTO documents_vec (rowid, library_id, version_id, embedding)
	SELECT d.id, v.library_id, v.id, json_extract(d.embedding, '$')
	FROM documents d
	JOIN pages p ON d.page_id = p.id
	JOIN versions v ON p.version_id = v.id
	WHERE d.embedding IS NOT NULL;
	`);
	// Ensure that inspection, potential drop, creation, and backfill are atomic
	this.db.exec("BEGIN IMMEDIATE;");
	try {
	const existingSql = this.db
	.prepare(
	"SELECT sql FROM sqlite_master WHERE type = 'table' AND name = 'documents_vec';",
	)
	.get() as { sql: string } | undefined;
	if (existingSql) {
	const match = existingSql.sql.match(/embedding\s+FLOAT\s*\[\s*(\d+)\s*]/i);
	const existingDim = match ? Number(match[1]) : null;
	if (existingDim === dim) {
	this.db.exec("COMMIT;");
	return;
	}
	this.db.exec("DROP TABLE IF EXISTS documents_vec;");
	}
	this.db.exec(`
	CREATE VIRTUAL TABLE IF NOT EXISTS documents_vec USING vec0(
	library_id INTEGER NOT NULL,
	version_id INTEGER NOT NULL,
	embedding FLOAT[${dim}]
	);
	`);
	this.db.exec(`
	INSERT OR REPLACE INTO documents_vec (rowid, library_id, version_id, embedding)
	SELECT d.id, v.library_id, v.id, json_extract(d.embedding, '$')
	FROM documents d
	JOIN pages p ON d.page_id = p.id
	JOIN versions v ON p.version_id = v.id
	WHERE d.embedding IS NOT NULL;
	`);
	this.db.exec("COMMIT;");
	} catch (error) {
	this.db.exec("ROLLBACK;");
	throw error;
	}

[Copilot]

Dropping documents_vec unconditionally forces a full vector index rebuild on next startup for every existing installation (even when staying on 1536d). For large stores this can be a costly operation and increase downtime. If runtime reconciliation already handles mismatched dimensions, consider avoiding the unconditional drop here (or limiting it to cases where it’s actually needed) to preserve existing indexes when the configured dimension hasn’t changed.

[Copilot]

New runtime behavior (creating/reconciling documents_vec based on embeddings.vectorDimension) doesn’t appear to have direct test coverage. Adding tests around DocumentStore initialization that assert the created table schema uses the configured dimension, and that changing the configured dimension is handled safely (including existing embeddings) would help prevent regressions.