Add GitHub Copilot extension

[waldekmastykarz]

Extended Dev Proxy Toolkit with a GitHub Copilot extension which would allow to generate proxy presets using natural language. The custom agent is grounded in Dev Proxy docs and knows what's possible and how to set it up.

[garrytrinder]

Awesome idea!

[Adam-it]

also you may use the LLM integration in any part of VS Code so for instance you could create a command or an action that is active only when someone has devproxy config JSON or mock JSON file opened that could allow to explain the contents of the file but explain it in a way that is valid and correlates to devproxy documentation. So not something generic like "you see a json config file with plugins section.. etc" but the aim would be to get something like "currently you configured devproxy to generate mocks based on intercepted requests and to generate HTTP file without including OPTIONS requests"

[waldekmastykarz]

That's a great idea @Adam-it! Currently, there's a limitation when it comes to bringing custom knowledge to ground the responses in factual and up-to-date information. We're exploring different ways of doing RAG using local SLMs for this. We've also suggested that VSCode exposes a new API for generating embeddings to support such scenarios.

[Adam-it]

That would be helpful indeed.
Currently to accommodate this kind of behavior I use chat tool that under the hood uses CLI for M365 to query that part of SPO and then the LLM analysis the response

[waldekmastykarz]

What do you mean with to query that part of SPO?

[Adam-it]

What do you mean with to query that part of SPO?

Sorry for not being clear. I am not sure if you know how it works in SPFx Toolkit. You may check this recording for more context

spfx-chat-movie1.mp4

What I mean by this is:

• user ask a questions about his/hers SPO tenant like what app catalogs do I have
• the LLM understands CLI for M365 commands and picks the best command to use for the user query based on command description and examples.
• the LLM checks if all required options are available in user prompt and if not starts a loop with prompts to ask for options
• if LLM has everything it presents the command and
• then a GH Copilot uses a tool to run the command under the hood and parse the response
• then a LLM with different grounding context analysis the response from the CLI for M365 command and explains the response to the user in natural language to the user

[waldekmastykarz]

the LLM understands CLI for M365 commands and picks the best command to use for the user query based on command description and examples

What does the LLM use to understand CLI's commands? Do you have a list of all commands and their descriptions and feed that into the LLM context along with the prompt?

How do you provide grounding context to the LLM?

[Adam-it]

What does the LLM use to understand CLI's commands? Do you have a list of all commands and their descriptions and feed that into the LLM context along with the prompt?

Yes

How do you provide grounding context to the LLM?

As part of the prompt as part of assistant along with history. I plan to refactor it to use prompt-tsx npm package which will simplify this a lot but I have some trouble with combining this along with react npm packages.

@waldekmastykarz if you are interested in more we may have a short call to catch up and I may go over the code with you. It's actually quite simple.

[waldekmastykarz]

I think I understand it. The biggest challenge is the context window size. While it might be enough to fit the list of commands and their descriptions, it's too small to include the full docs right? This is exactly why we're looking at RAG scenarios to be able to search across full docs but then include the relevant sections/pages in the context window for a human-friendly response.

[Adam-it]

I think I understand it. The biggest challenge is the context window size. While it might be enough to fit the list of commands and their descriptions, it's too small to include the full docs right? This is exactly why we're looking at RAG scenarios to be able to search across full docs but then include the relevant sections/pages in the context window for a human-friendly response.

yes. Currently I managed to get all list and get commands along with full description for each option and examples. Its like 112 commands so that is already stretching the token limit which in case of GPT4-o which I use is 4k.
As for cases similar to RAG what is already possible which I also considered (and still consider or partly use) are:

• you may implement your own GitHub Copilot extension and this may work also across many different places/IDESs , like github.com and also VS Code. The problem is that everything is on your side so you need to host the LLM together with custom logic like for example on Azure and expose it over API and then register as an GitHub app. It may also not interact with VS Code specific API, it may just be a source of knowledge for prompts
• you may also LanguageModelTool API. This allows you to register list of tools that your chat participant may use to ask for additional context or to take/trigger some action. The problem is that the 'tool' usually has to be either a separate VS Code extension or an API which you need to store/handle and therefore maintain and usually pay for some storage or usage depending which cloud provider would you pick.... or keep your laptop always on 😜.

have you considered any of those for your scenario?

[waldekmastykarz]

We're looking at alternatives to avoid operating an API and picking up the tab for costs. This is exactly why we're looking at local RAG scenarios using things like cosine similarity on embeddings to find the relevant content from the docs. It however requires the ability to get embeddings for the user's prompt which in turn requires a language model for embeddings, which right now isn't available through the GitHub/VSCode APIs. To showcase the scenario, we could use Ollama, but for the best experience, we'd need an API.

[Adam-it]

Yeah I get it. Searching for free and open solution was also one of the reasons why I picked my approach

[garrytrinder]

Closing as we now include the Dev Proxy MCP server in the latest beta release. #267