feat: add provider API for listing and inspecting provider info by cdoern · Pull Request #1429 · llamastack/llama-stack

cdoern · 2025-03-05T21:01:01Z

What does this PR do?

currently the inspect API for providers is really a list API. Create a new providers API which has a GET providers/{provider_id} inspect API
which returns "user friendly" configuration to the end user. Also add a GET /providers endpoint which returns the list of providers as inspect/providers does today.

This API follows CRUD and is more intuitive/RESTful.

This work is part of the RFC at #1359

sensitive fields are redacted using redact_sensetive_fields on the server side before returning a response:

Test Plan

using llamastack/llama-stack-client-python#181 a user is able to to run the following:

llama stack build --template ollama --image-type venv
llama stack run --image-type venv ~/.llama/distributions/ollama/ollama-run.yaml
llama-stack-client providers inspect ollama

also, was able to run the new test_list integration test locally with ollama:

raghotham · 2025-03-05T23:27:33Z

Unless I'm missing something, seems like we should try to consolidate the work in #1359 and here? Maybe we should just have a /providers api with all CRUD operations?

cdoern · 2025-03-06T01:40:58Z

@raghotham Sorry about the churn, do you prefer I use the RFC PR for implementation as well? I did two separate PRs (one to introduce the RFC doc and another to do some implementation of the ideas int he RFC) but can consolidate the PRs if necessary! let me know 🚀

raghotham

Thanks for making the change. Added a couple of comments. Also, guessing you will be adding the create/update methods in a different PR.

raghotham · 2025-03-10T19:35:32Z

llama_stack/apis/providers/providers.py

maybe this can just be a getProviderResponse?

raghotham · 2025-03-10T19:38:28Z

llama_stack/distribution/datatypes.py

not very clear we need UserConfig. There's already a SecretStr type so that we dont return secrets

@raghotham yeah, I could use that. I was thinking UserConfig could be useful for really explicitly choosing what parts of the configuration we want to expose to the user rather than showing everything and just hiding parts with SecretStr

Can we add UserConfig as a next step? For now, expose everything other than SecretStr?

sure yeah, fine by me!

llama_stack/distribution/configure.py

raghotham · 2025-03-13T12:35:35Z

@cdoern would you be able to get it to a mergeable state in time for 0.1.7 release cut EOD today?

cdoern · 2025-03-13T12:41:00Z

@raghotham yep, I have some work locally I can push up within an hour

ashwinb · 2025-03-13T18:58:05Z

llama_stack/distribution/providers.py

we probably need a fast path to such builtin APIs rather than having to go through the whole resolution abstraction (since there never will be multiple "implementations" of this API)

yep I agree, maybe we should scope future work to add this for inspect and provider APIs?

ashwinb · 2025-03-13T18:59:14Z

llama_stack/providers/remote/inference/ollama/config.py

why is this added only here? is this annotation not relevant to other places?

apologies, leftover, thanks for catching this! Removed

cdoern · 2025-03-13T19:35:09Z

integration tests will fail due to pending client changes since /v1/inspect/providers is still being used in integration tests.

This PR should fix it and should be merged with this PR: llamastack/llama-stack-client-python#181

cdoern · 2025-03-13T19:41:37Z

image of integration tests passing with llamastack/llama-stack-client-python#181 installed:

currently the `inspect` API for providers is really a `list` API. Create a new `providers` API which has a GET `providers/{provider_id}` inspect API which returns "user friendly" configuration to the end user. Also add a GET `/providers` endpoint which returns the list of providers as `inspect/providers` does today. This API follows CRUD and is more intuitive/RESTful. This work is part of the RFC at llamastack#1359 Signed-off-by: Charlie Doern <cdoern@redhat.com>

cdoern · 2025-03-13T21:56:23Z

please note, I kept /v1/inspect/providers for compatibility but future work to deprecate this API should be planned!

ashwinb · 2025-03-13T22:06:25Z

@cdoern could you file an issue for deprecation with like a 2 week target date so we don't forget?

# What does this PR do? allow a user to see certain parts of the provider configuration for a specified provider. This is the cli code corresponding to llamastack/llama-stack#1429 This PR introduces: - `llama-stack-client providers inspect` - amends `llama-stack-client providers list` to use `/v1/providers` - `GetProviderResponse` to handle the response from these new API calls Signed-off-by: Charlie Doern <cdoern@redhat.com>

yanxi0830 · 2025-03-14T00:09:06Z

llama_stack/apis/providers/providers.py

+    async def list_providers(self) -> ListProvidersResponse: ...
+
+    @webmethod(route="/providers/{provider_id}", method="GET")
+    async def inspect_provider(self, provider_id: str) -> GetProviderResponse: ...


shouldn't this return ProviderInfo?

ProviderInfo doesn't contain a config, so I created a new type for that. inspect returns detailed info about a provider. list returns ProviderInfo

@cdoern

# What does this PR do? - #1429 introduces GetProviderResponse in OpenAPI, which is not needed, and not correctly defined. cc @cdoern [//]: # (If resolving an issue, uncomment and update the line below) [//]: # (Closes #[issue-number]) ## Test Plan ``` llama-stack-client providers list ``` <img width="610" alt="image" src="https://github.com/user-attachments/assets/2f7b62a5-daf2-4bf9-9505-69755c7025fc" /> [//]: # (## Documentation)

add `v1/providers/` which uses PUT to allow users to change their provider configuration this is a follow up to llamastack#1429 and related to llamastack#1359 a user can call something like: `llama_stack_client.providers.update(api="inference", provider_id="ollama", provider_type="remote::ollama", config={'url': 'http:/localhost:12345'})` or `llama-stack-client providers update inference ollama remote::ollama "{'url': 'http://localhost:12345'}"` this API works by adding a `RequestMiddleware` to the server which checks requests, and if the user is using PUT /v1/providers, the routes are re-registered with the re-initialized provider configurations/methods for the client, `self.impls` is updated to hold the proper methods+configurations this depends on a client PR, the CI will fail until then but succeeded locally Signed-off-by: Charlie Doern <cdoern@redhat.com>

facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Mar 5, 2025

cdoern force-pushed the config-api branch 3 times, most recently from 75b1bdf to 368be3e Compare March 10, 2025 19:10

cdoern changed the title ~~feat: add inspect API for user-friendly provider config~~ feat: add provider API for listing and inspecting provider info Mar 10, 2025

raghotham reviewed Mar 10, 2025

View reviewed changes

jaideepr97 reviewed Mar 12, 2025

View reviewed changes

llama_stack/distribution/configure.py Outdated Show resolved Hide resolved

raghotham added this to the v0.1.7 milestone Mar 13, 2025

cdoern force-pushed the config-api branch from 368be3e to 72dfb39 Compare March 13, 2025 14:44

cdoern marked this pull request as ready for review March 13, 2025 14:44

cdoern requested review from SLR722, ashwinb, dineshyv, dltn, ehhuang, hardikjshah, sixianyi0721, terrytangyuan, vladimirivic and yanxi0830 as code owners March 13, 2025 14:44

cdoern force-pushed the config-api branch 5 times, most recently from 0edb668 to eeda561 Compare March 13, 2025 15:14

cdoern requested a review from raghotham March 13, 2025 18:29

ashwinb reviewed Mar 13, 2025

View reviewed changes

cdoern force-pushed the config-api branch from eeda561 to 22076d5 Compare March 13, 2025 19:14

raghotham mentioned this pull request Mar 13, 2025

feat: llama-stack-client providers inspect PROVIDER_ID llamastack/llama-stack-client-python#181

Merged

raghotham approved these changes Mar 13, 2025

View reviewed changes

cdoern force-pushed the config-api branch from 22076d5 to 520b550 Compare March 13, 2025 21:40

ashwinb merged commit a062723 into llamastack:main Mar 13, 2025
9 checks passed

cdoern mentioned this pull request Mar 13, 2025

Deprecate /v1/inspect/providers #1623

Closed

yanxi0830 reviewed Mar 14, 2025

View reviewed changes

yanxi0830 mentioned this pull request Mar 14, 2025

fix: OpenAPI with provider get #1627

Merged

cdoern mentioned this pull request Apr 8, 2025

feat: implement provider updating #1905

Closed

Conversation

cdoern commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Test Plan

Uh oh!

raghotham commented Mar 5, 2025

Uh oh!

cdoern commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

raghotham left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

raghotham commented Mar 13, 2025

Uh oh!

cdoern commented Mar 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cdoern commented Mar 13, 2025

Uh oh!

cdoern commented Mar 13, 2025

Uh oh!

cdoern commented Mar 13, 2025

Uh oh!

ashwinb commented Mar 13, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

cdoern commented Mar 5, 2025 •

edited

Loading

cdoern commented Mar 6, 2025 •

edited

Loading