feat(cli,nvim): add nix flake and option to specify cli_cmds in nvim #144

chrishrb · 2025-05-16T07:03:00Z

This PR introduces a Nix flake. To ensure Nix works correctly, it also updates the Neovim plugin to support specifying cli_cmds, which are later patched by Nix to point directly to the Nix store. Both changes are included in this PR.

codecov · 2025-05-16T07:05:08Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.27%. Comparing base (2ed7d79) to head (fec8d6d).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #144   +/-   ##
=======================================
  Coverage   99.27%   99.27%           
=======================================
  Files          22       22           
  Lines        1519     1523    +4     
=======================================
+ Hits         1508     1512    +4     
  Misses         11       11

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Davidyz · 2025-05-16T10:16:06Z

Hi, thanks for this PR! I've never personally used Nix, so I wouldn't be able to maintain the Nix-related code. I have some questions related to the maintenance of the Nix stuff:

Would the nix-related files need updating when I make a new release?
If they do, would you be able to help maintain them?
There was an issue related to the Nix package: [BUG] vectorcode ls fails #73 . Would your PR help fix this issue?

I'll finish reviewing the rest of the PR soon. Thanks again for doing this!

chrishrb · 2025-05-16T13:20:06Z

you should definitely try it 😄 it's the same rabbit hole as neovim 😄

yea thats a problem. I would create a nixpkgs but chromadb is only on version 0.5.18, but vectorcode needs 0.6.x
yes, at least until chromadb is upgraded in nixpkgs
yes. That was related to the chromadb version, which was only 0.5.18 but we need at least 0.6.x

If I have the time I try to upgrade chromadb in nixpkgs also so I can create the package for vectorcode. Do you know if vectorcode also supports chromadb 1.x?

Davidyz · 2025-05-17T01:24:33Z

yea thats a problem. I would create a nixpkgs but chromadb is only on version 0.5.18, but vectorcode needs 0.6.x

Does this mean the current nix package builds chroma 0.6.3 from source? I see there are references to cargo, is that used for building chroma-hnswlib (this used to cause problems for ppl without the build deps so I want to know whether this is taken care of)?

Do you know if vectorcode also supports chromadb 1.x?

The Chromadb problem is a bit complicated. I've mostly done the migration to chromadb 1.0.x in #36, and on a clean installation (for both chroma and vectorcode) it works, but if the database files were created by chroma 0.6.x it doesn't work, and it's an upstream bug. Therefore, I won't be bumping the chroma version to 1.0.x any time soon (unless they fix the incompatibility issue), and when that happens it'll be marked as a breaking change (most likely with VectorCode 1.0.0).

Davidyz

Sorry about delaying this for so long. Since this PR contains quite some changes, I thought I'd make a dedicated release for it so that if anything goes wrong, people can easily revert to previous versions without losing other features/fixes.

The changes are mostly good to me, except that I don't think it's necessary to make the LSP executable configurable through setup(). Although your approach is technically ok, it adds an extra source of confusion over "which option takes precedence", so I think it's better to keep it to one place and just use vim.lsp.config.

Another thing that is missing from your PR is the content in lua/vectorcode/init.lua, which also contains quite a few hardcoded vectorcode executable. I'll address this in a separate PR (replace them with jobrunner) and merge it before this one, so that all cmd calls correctly use the configured executable.

Davidyz · 2025-05-20T11:48:38Z

docs/neovim.md

 require("vectorcode").setup({
+  cli_cmds = {
+    vectorcode = "vectorcode",
+    vectorcode_server = "vectorcode-server",


We can get rid of vectorcode_server here because the cmd is loaded from vim.lsp.config. In other words, it's already possible to overload the executable for the LSP runner.

Davidyz · 2025-05-21T10:09:55Z

I've rebased changes to init.lua into this branch. All hardcoded paths should've been replaced.

Next, I'll try to set up Nix and do some final tests.

Davidyz · 2025-05-22T04:16:22Z

btw I'd like to add a formatter for the nix flake (either pre-commit or github action). Do you have any recommendations?

willruggiano · 2025-05-23T07:48:59Z

btw I'd like to add a formatter for the nix flake (either pre-commit or github action). Do you have any recommendations?

alejandra is a popular one. Also check out treefmt. The nixpkgs repo itself uses nixfmt-rfc-style but I personally find its style less appealing.

Not that anyone asked for my opinion but I would also recommend against hardcoding the executable paths like this. IMO it adds unnecessary complexity (see: the majority of code changes in this PR) when people can just as easily manipulate their PATH. Conversely, one might even say that "vectorcode" is not hardcoding at all.

Davidyz · 2025-05-29T10:43:59Z

I was able to run vectorise and query commands in nix shell, but ls doesn't work. It's the same error as #73 . @chrishrb do you know why this is happening? Also, am I supposed to be using a standalone chromadb or the bundled chromadb (not sure if this is related tbh)?

willruggiano · 2025-05-29T21:34:53Z

@Davidyz when you say "run ... in nix shell" what do you mean? As it stands on this branch, I'm not sure the nix-shell will "do" anything besides setup some environment variables (PYTHONNOUSERSITE=1 and PYTHONPATH=/nix/store/...) which, most notably, enables tools like basedpyright, ruff, etc to work properly against the codebase. In particular, inside of a nix-shell you should not have a vectorcode command (the shell did not build the package, just the deps) and things like make deps will fail ¹ because the nix-shell does not put pdm on your PATH (at least in a pure Nix setup that is).

Perhaps a better test is to nix build and then run via ./result/bin/vectorcode (or just nix run).

Note that I have experimented with this branch in all sorts of way ² and haven't been able to reproduce this error.

$ make deps
pdm lock --group dev --group lsp --group mcp; \
pdm install
sh: line 1: pdm: command not found
sh: line 2: pdm: command not found
make: *** [Makefile:4: deps] Error 127

↩

mainly with chromadb-0.5.20 (via nixpkgs; running as a systemd service) and chromadb-0.6.3 (via this branch i.e. bundled) ↩

Davidyz · 2025-05-30T10:09:44Z

@Davidyz when you say "run ... in nix shell" what do you mean? As it stands on this branch, I'm not sure the nix-shell will "do" anything besides setup some environment variables (PYTHONNOUSERSITE=1 and PYTHONPATH=/nix/store/...) which, most notably, enables tools like basedpyright, ruff, etc to work properly against the codebase. In particular, inside of a nix-shell you should not have a vectorcode command (the shell did not build the package, just the deps) and things like make deps will fail 1 because the nix-shell does not put pdm on your PATH (at least in a pure Nix setup that is).

Perhaps a better test is to nix build and then run via ./result/bin/vectorcode (or just nix run).

Note that I have experimented with this branch in all sorts of way 2 and haven't been able to reproduce this error.

Footnotes
1. ```
   $ make deps
   pdm lock --group dev --group lsp --group mcp; \
   pdm install
   sh: line 1: pdm: command not found
   sh: line 2: pdm: command not found
   make: *** [Makefile:4: deps] Error 127
   ```
   
   
   [↩](#user-content-fnref-1-d1debe5732acd3d2582d453415e19cab)

2. mainly with chromadb-0.5.20 (via nixpkgs; running as a systemd service) and chromadb-0.6.3 (via this branch i.e. bundled) [↩](#user-content-fnref-2-d1debe5732acd3d2582d453415e19cab)

Thanks for the suggestions. After some more research on Nix, I'm pretty sure it was because I didn't RTFM enough. I'm now able to work with the CLI with no problems. There are still some things to fix before I merge this, but I now have a pretty good idea about what to do next.

betaboon · 2025-05-31T12:29:44Z

@chrishrb as a fellow nix user, could the flake also contain an overlay? that would make consumption even easier.

betaboon · 2025-05-31T12:47:20Z

flake.nix

+          };
+        };
+
+        pkgVersion = "0.5.6";


this is kinda confusing as it shows up in vectorcode --help as its version.
i don't have a better idea from the top of my head tho.

maybe just nightly ?

I plan to change this to the most recent version just before merging. As far as I understand, it's not recommended in Nix to use a non-deterministic/non-reproducible version number, so we'll probably need to bump pkgVersion for every release.

sarahec · 2025-05-31T16:54:22Z

vectorcode is already in nixpkgs (which you know because you copied the chromadb derivation from there). There's no need for a separate flake here, especially if our gracious host won't be able to maintain it.

If you needed chromadb to be at 0.6.x, why not file an upgrade ticket at NixOS/nixpkgs?

P.S. As long as I'm bearing bad news, that one commit really should be multiple logical commits (doc update, add flake, adding cli_cmds for nvim, etc.).

Davidyz · 2025-06-02T02:03:45Z

that one commit really should be multiple logical commits

I've been thinking about this too. I plan to checkout from this PR, strip all the nix-related stuff and merge it in a new PR. This'll address the hardcoded "vectorcode" executables from the nvim plugin, and should make lives easier for nix folks. I can also mention the nixpkg in the documentation. The nix flake, if still necessary, can have its own PR.

sarahec · 2025-06-02T02:52:17Z

I've been thinking about this too. I plan to checkout from this PR, strip all the nix-related stuff and merge it in a new PR.

FYI, vectorcode is fully supported in nix and is kept current. I forked chromadb into python3Packages.chromadb_0 @ 0.6.3 and python3Packages.chromadb is heading for 1.0.12 in NixOS/nixpkgs#412528

sarahec · 2025-06-02T19:35:01Z

flake.nix

+          vimPlugin = pkgs.vimUtils.buildVimPlugin {
+            pname = "VectorCode";
+            version = pkgVersion;
+            src = self;
+            dependencies = [
+              pkgs.vimPlugins.plenary-nvim
+            ];
+            patches = pkgs.replaceVars ./nix/vim-plugin.patch {
+              inherit vectorcode;
+            };


Wouldn't this make more sense as its own derivation in nixpkgs? There's already over 2000 vim plugins avalilable.

This should be an easy PR.

sarahec · 2025-06-02T22:15:55Z

Added the Neovim plugin to Nixpkgs for you: NixOS/nixpkgs#413395

It gets around the cli_cmds requirement by packaging vectorcode inside the plugin. (Nix packages are bundled with all of their dependencies, in other words.)

sarahec · 2025-06-02T22:35:27Z

I was able to run vectorise and query commands in nix shell, but ls doesn't work. It's the same error as #73 . @chrishrb do you know why this is happening? Also, am I supposed to be using a standalone chromadb or the bundled chromadb (not sure if this is related tbh)?

chromadb was neglected for a while and was sitting at 0.5.20. I've filed a PR to bring that up to 0.6.3 for vectorcode's use and 1.0.12 for everybody else.

I'm an active maintainer, so I'll stay on top of this.

sarahec · 2025-06-02T22:41:18Z

@chrisrb I have nixpkgs PRs open to bring chromadb up to 0.6.3 (NixOS/nixpkgs#412528) and to add the neovim plugin (NixOS/nixpkgs#413395). Will those two changes meet your needs?

chrishrb · 2025-06-03T19:18:00Z

@sarahec thanks a lot! this is an even better solution. for me this PR isn't necessary anymore now 👍

sarahec · 2025-06-03T22:46:44Z

@sarahec thanks a lot! this is an even better solution. for me this PR isn't necessary anymore now 👍

Excellent! Now all I have to do is get the chromadb update (to 0.6.3) merged at nixpkgs and you'll be all set.

Davidyz · 2025-06-04T05:01:50Z

@sarahec thanks a lot! this is an even better solution. for me this PR isn't necessary anymore now 👍

In this case I'll close this PR in favour of #170 (which builds on top of this PR anyway). Thanks for your work!

chrishrb force-pushed the feature/nix-flake branch from dae5a09 to 2fe9869 Compare May 16, 2025 07:05

Davidyz mentioned this pull request May 19, 2025

[ENH] Neovim plugin: make path to vectorcode executable configurable #148

Closed

Davidyz linked an issue May 19, 2025 that may be closed by this pull request

[ENH] Neovim plugin: make path to vectorcode executable configurable #148

Closed

Davidyz requested changes May 20, 2025

View reviewed changes

Davidyz added enhancement New feature or request feature labels May 20, 2025

Davidyz mentioned this pull request May 21, 2025

refactor(nvim): refactor more APIs to use jobrunner. #155

Merged

Davidyz force-pushed the feature/nix-flake branch 2 times, most recently from f678e3e to 9968537 Compare May 21, 2025 10:06

Davidyz force-pushed the feature/nix-flake branch from 9968537 to 13d44d6 Compare May 29, 2025 09:38

Davidyz force-pushed the feature/nix-flake branch from e3aa67b to 868a1d4 Compare May 29, 2025 11:14

betaboon reviewed May 31, 2025

View reviewed changes

feat(cli,nvim): add nix flake and option to specify cli_cmds in nvim

fec8d6d

Davidyz force-pushed the feature/nix-flake branch from 868a1d4 to fec8d6d Compare June 2, 2025 01:53

sarahec reviewed Jun 2, 2025

View reviewed changes

Davidyz mentioned this pull request Jun 3, 2025

Configurable executable path for the nvim plugin #170

Merged

Davidyz closed this Jun 4, 2025

Davidyz mentioned this pull request Jun 17, 2025

[BUG] Chromadb client and sever version mismatches cause a KeyError when running vectorcode. #194

Closed

Uh oh!

feat(cli,nvim): add nix flake and option to specify cli_cmds in nvim #144

feat(cli,nvim): add nix flake and option to specify cli_cmds in nvim #144

Uh oh!

Conversation

chrishrb commented May 16, 2025

Uh oh!

codecov bot commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Davidyz commented May 16, 2025

Uh oh!

chrishrb commented May 16, 2025

Uh oh!

Davidyz commented May 17, 2025

Uh oh!

Davidyz left a comment

Choose a reason for hiding this comment

Uh oh!

Davidyz May 20, 2025

Choose a reason for hiding this comment

Uh oh!

Davidyz commented May 21, 2025

Uh oh!

Davidyz commented May 22, 2025

Uh oh!

willruggiano commented May 23, 2025

Uh oh!

Davidyz commented May 29, 2025

Uh oh!

willruggiano commented May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Footnotes

Uh oh!

Davidyz commented May 30, 2025

Footnotes

Uh oh!

betaboon commented May 31, 2025

Uh oh!

betaboon May 31, 2025

Choose a reason for hiding this comment

Uh oh!

Davidyz May 31, 2025

Choose a reason for hiding this comment

Uh oh!

sarahec commented May 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Davidyz commented Jun 2, 2025

Uh oh!

sarahec commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sarahec Jun 2, 2025

Choose a reason for hiding this comment

Uh oh!

sarahec Jun 2, 2025

Choose a reason for hiding this comment

Uh oh!

sarahec commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sarahec commented Jun 2, 2025

Uh oh!

sarahec commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chrishrb commented Jun 3, 2025

Uh oh!

sarahec commented Jun 3, 2025

Uh oh!

Davidyz commented Jun 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

codecov bot commented May 16, 2025 •

edited

Loading

willruggiano commented May 29, 2025 •

edited

Loading

sarahec commented May 31, 2025 •

edited

Loading

sarahec commented Jun 2, 2025 •

edited

Loading

sarahec commented Jun 2, 2025 •

edited

Loading

sarahec commented Jun 2, 2025 •

edited

Loading