feat(nvim): avoid retrieving files that is already present in the context

[Davidyz]

This PR adds an option that check the existing references that have been added to the chat, and excluding them from further queries. This helps saving tokens, and should help the search if previous queries missed the appropriate documents.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.78%. Comparing base (16e78ed) to head (d782e78).
Report is 2 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main      #72   +/-   ##
=======================================
  Coverage   97.78%   97.78%           
=======================================
  Files          18       18           
  Lines        1176     1176           
=======================================
  Hits         1150     1150           
  Misses         26       26           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Davidyz]

@olimorris Could you take a look at this when you're available? I'm not sure whether there's a better way to do this.

EDIT: specifically whether it's possible to simplify this by making use of the chat.references class (I want to be able to de-duplicate using the --exclude flag while allowing retrieval results from previous calls to stay).

[olimorris]

The chat.references table should have the path for each file or buffer that's been shared, if I recall (sorry on mobile right now).

I'd recommend using the references in the chat object simply because it's a live reflection of what context the user has manually added.

[Davidyz]

I'm not quite sure how to make it work with references, though. If I create a new CodeCompanion.Chat.Ref object for each file, it'll populate the reference block in the chat UI very quickly, especially if you query for a large number of files or make multiple calls in one chat. If all vectorcode context goes into the same Ref object, it'll be tricky to get the paths for files for deduplication because the content field for a message needs to be a string.

[Davidyz]

This is occupying half of the height of the window... Looks a bit overwhelming lol

Related PR: olimorris/codecompanion.nvim#1245

[olimorris]

I'm not quite sure how to make it work with references, though. If I create a new CodeCompanion.Chat.Ref object for each file, it'll populate the reference block in the chat UI very quickly, especially if you query for a large number of files or make multiple calls in one chat. If all vectorcode context goes into the same Ref object, it'll be tricky to get the paths for files for deduplication because the content field for a message needs to be a string.

Apologies for coming to this late...does your PR resolve this?

One thought I had was that if I added an event in CodeCompanion that fires when a reference is added, this would allow VectorCode to stay in sync with the chat buffer. Now the downside here is that if VectorCode is added after the fact then there would still be that initial processing to do.

OR...is there a more useful public facing references table that could be created on the chat buffer class?

[Davidyz]

does your PR resolve this?

My PR to codecompanion allows finer controls over whether a particular Ref should be displayed, so I can hide the vectorcode references so that the vectorcode files won't occupy the whole height of the window.

One thought I had was that if I added an event in CodeCompanion that fires when a reference is added, this would allow VectorCode to stay in sync with the chat buffer.

This may be useful if I want to --exclude files and buffers added by slash commands or variables. But I'd still need to check for the chat.refs list at the beginning of the tool function anyways, in case the @vectorcode tool is not the first reference to be added.

OR...is there a more useful public facing references table that could be created on the chat buffer class?

From my side the current reference system works fine. I still need to experiment on the pinned and watched options, but in general it works for what I'm doing.