Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 104 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
<div align="center">
<picture align="center">
<source media="(prefers-color-scheme: dark)" srcset="media/icons/ai_studio_icon_white.svg">
<source media="(prefers-color-scheme: light)" srcset="media/icons/ai_studio_icon_black.svg">
<img alt="AI Foundry icon." src="media/icons/ai_studio_icon_black.svg" height="100" style="max-width: 100%;">
<source media="(prefers-color-scheme: dark)" srcset="media/icons/foundry_local_white.svg">
<source media="(prefers-color-scheme: light)" srcset="media/icons/foundry_local_black.svg">
<img alt="AI Foundry icon." src="media/icons/foundry_local_black.svg" height="100" style="max-width: 100%;">
</picture>
<div id="user-content-toc">
<ul align="center" style="list-style: none;">
Expand Down Expand Up @@ -69,7 +69,107 @@ This will show you a list of all models that can be run locally, including their

## 🧑‍💻 Integrate with your applications using the SDK

Foundry Local has an easy-to-use SDK (Python, JavaScript) to get you started with existing applications:
Foundry Local has an easy-to-use SDK (C#, Python, JavaScript) to get you started with existing applications:

### C#

The C# SDK is available as a package on NuGet. You can install it using the .NET CLI:

```bash
dotnet add package Microsoft.AI.Foundry.Local.WinML
```

> [!TIP]
> The C# SDK does not require end users to have Foundry Local CLI installed. It is a completely self-contained SDK that will does not depend on any external services. Also, the C# SDK has native in-process Chat Completions and Audio Transcription APIs that do not require HTTP calls to the local Foundry service.

Here is an example of using the C# SDK to run a model and generate a chat completion:

```csharp
using Microsoft.AI.Foundry.Local;
using Betalgo.Ranul.OpenAI.ObjectModels.RequestModels;
using Microsoft.Extensions.Logging;

CancellationToken ct = new CancellationToken();

var config = new Configuration
{
AppName = "my-app-name",
LogLevel = Microsoft.AI.Foundry.Local.LogLevel.Debug
};

using var loggerFactory = LoggerFactory.Create(builder =>
{
builder.SetMinimumLevel(Microsoft.Extensions.Logging.LogLevel.Debug);
});
var logger = loggerFactory.CreateLogger<Program>();

// Initialize the singleton instance.
await FoundryLocalManager.CreateAsync(config, logger);
var mgr = FoundryLocalManager.Instance;

// Get the model catalog
var catalog = await mgr.GetCatalogAsync();

// List available models
Console.WriteLine("Available models for your hardware:");
var models = await catalog.ListModelsAsync();
foreach (var availableModel in models)
{
foreach (var variant in availableModel.Variants)
{
Console.WriteLine($" - Alias: {variant.Alias} (Id: {string.Join(", ", variant.Id)})");
}
}

// Get a model using an alias
var model = await catalog.GetModelAsync("qwen2.5-0.5b") ?? throw new Exception("Model not found");


// is model cached
Console.WriteLine($"Is model cached: {await model.IsCachedAsync()}");

// print out cached models
var cachedModels = await catalog.GetCachedModelsAsync();
Console.WriteLine("Cached models:");
foreach (var cachedModel in cachedModels)
{
Console.WriteLine($"- {cachedModel.Alias} ({cachedModel.Id})");
}

// Download the model (the method skips download if already cached)
await model.DownloadAsync(progress =>
{
Console.Write($"\rDownloading model: {progress:F2}%");
if (progress >= 100f)
{
Console.WriteLine();
}
});

// Load the model
await model.LoadAsync();

// Get a chat client
var chatClient = await model.GetChatClientAsync();

// Create a chat message
List<ChatMessage> messages = new()
{
new ChatMessage { Role = "user", Content = "Why is the sky blue?" }
};

var streamingResponse = chatClient.CompleteChatStreamingAsync(messages, ct);
await foreach (var chunk in streamingResponse)
{
Console.Write(chunk.Choices[0].Message.Content);
Console.Out.Flush();
}
Console.WriteLine();

// Tidy up - unload the model
await model.UnloadAsync();
```


### Python

Expand Down
97 changes: 4 additions & 93 deletions docs/README.md
Original file line number Diff line number Diff line change
@@ -1,96 +1,7 @@
# Get Started with Foundry Local
# Foundry Local Documentation

This guide provides detailed instructions on installing, configuring, and using Foundry Local to run AI models on your device.
The Foundry Local documentation is provided on [Microsoft Learn](https://learn.microsoft.com/azure/ai-foundry/foundry-local/) where you can find comprehensive guides and tutorials to help you get started and make the most of the Foundry Local.

## Prerequisites
## API Reference

- A PC with sufficient specifications to run AI models locally
- Windows 10 or later
- Greater than 8GB RAM
- Greater than 10GB of free disk space for model caching (quantized Phi 3.2 models are ~3GB)
- Suggested hardware for optimal performance:
- Windows 11
- NVIDIA GPU (2000 series or newer) OR AMD GPU (6000 series or newer) OR Qualcomm Snapdragon X Elite, with 8GB or more of VRAM
- Greater than 16GB RAM
- Greater than 20GB of free disk space for model caching (the largest models are ~15GB)
- Administrator access to install software

## Installation

1. Download Foundry Local for your platform from the [releases page](https://github.com/microsoft/Foundry-Local/releases).
2. Install the package by following the on-screen prompts.
3. After installation, access the tool via command line with `foundry`.

## Running Your First Model

1. Open a command prompt or terminal window.
2. Run a model using the following command:

```bash
foundry model run phi-3.5-mini
```

This command will:

- Download the model to your local disk
- Load the model into your device
- Start a chat interface

**💡 TIP:** Replace `phi-3.5-mini` with any model from the catalog. Use `foundry model list` to see available models.

## Explore Foundry Local CLI commands

The foundry CLI is structured into several categories:

- **Model**: Commands related to managing and running models
- **Service**: Commands for managing the Foundry Local service
- **Cache**: Commands for managing the local cache where models are stored

To see all available commands, use the help option:

```bash
foundry --help
```

**💡 TIP:** For a complete reference of all available CLI commands and their usage, see the [Foundry Local CLI Reference](./reference/reference-cli.md)

## Integrating with Applications

Foundry Local provides an OpenAI-compatible REST API at `http://localhost:PORT/v1`.

- Note that the port will be dynamically assigned, so check the logs for the correct port.

### REST API Example

```bash
curl http://localhost:5273/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Phi-3.5-mini-instruct-generic-cpu",
"messages": [{"role": "user", "content": "What is the capital of France?"}],
"temperature": 0.7,
"max_tokens": 50
}'
```

Read about all the samples we have for various languages and platforms in the [Integrate with Inference SDKs](./how-to/integrate-with-inference-sdks.md) section.

## Troubleshooting

### Common Issues and Solutions

| Issue | Possible Cause | Solution |
| ----------------------- | --------------------------------------- | ----------------------------------------------------------------------------------------- |
| Slow inference | CPU-only model on large parameter count | Use GPU-optimized model variants when available |
| Model download failures | Network connectivity issues | Check your internet connection, try `foundry cache list` to verify cache state |
| Service won't start | Port conflicts or permission issues | Try `foundry service restart` or post an issue providing logs with `foundry zip-logsrock` |

For more information, see the [troubleshooting guide](./reference/reference-troubleshooting.md).

## Next Steps

- [Learn more about Foundry Local](./what-is-foundry-local.md)
- [Integrate with inferencing SDKs](./how-to/integrate-with-inference-sdks.md)
- [Compile models for Foundry Local](./how-to/compile-models-for-foundry-local.md)
- [Build a chat application](./tutorials/chat-application-with-open-web-ui.md)
- [Use Langchain](./tutorials/use-langchain-with-foundry-local.md)
- [Foundry Local C# SDK API Reference](./cs-api/Microsoft.AI.Foundry.Local.md)
56 changes: 56 additions & 0 deletions docs/cs-api/Microsoft.AI.Foundry.Local.Configuration.WebService.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# <a id="Microsoft_AI_Foundry_Local_Configuration_WebService"></a> Class Configuration.WebService

Namespace: [Microsoft.AI.Foundry.Local](Microsoft.AI.Foundry.Local.md)
Assembly: Microsoft.AI.Foundry.Local.dll

Configuration settings if the optional web service is used.

```csharp
public class Configuration.WebService
```

#### Inheritance

[object](https://learn.microsoft.com/dotnet/api/system.object) ←
[Configuration.WebService](Microsoft.AI.Foundry.Local.Configuration.WebService.md)

#### Inherited Members

[object.Equals\(object?\)](https://learn.microsoft.com/dotnet/api/system.object.equals\#system\-object\-equals\(system\-object\)),
[object.Equals\(object?, object?\)](https://learn.microsoft.com/dotnet/api/system.object.equals\#system\-object\-equals\(system\-object\-system\-object\)),
[object.GetHashCode\(\)](https://learn.microsoft.com/dotnet/api/system.object.gethashcode),
[object.GetType\(\)](https://learn.microsoft.com/dotnet/api/system.object.gettype),
[object.MemberwiseClone\(\)](https://learn.microsoft.com/dotnet/api/system.object.memberwiseclone),
[object.ReferenceEquals\(object?, object?\)](https://learn.microsoft.com/dotnet/api/system.object.referenceequals),
[object.ToString\(\)](https://learn.microsoft.com/dotnet/api/system.object.tostring)

## Properties

### <a id="Microsoft_AI_Foundry_Local_Configuration_WebService_ExternalUrl"></a> ExternalUrl

If the web service is running in a separate process, it will be accessed using this URI.
Both processes should be using the same version of the SDK. If a random port is assigned when creating
the web service in the external process the actual port must be provided here.

```csharp
public Uri? ExternalUrl { get; init; }
```

#### Property Value

[Uri](https://learn.microsoft.com/dotnet/api/system.uri)?

### <a id="Microsoft_AI_Foundry_Local_Configuration_WebService_Urls"></a> Urls

Url/s to bind to the web service when <xref href="Microsoft.AI.Foundry.Local.FoundryLocalManager.StartWebServiceAsync(System.Nullable%7bSystem.Threading.CancellationToken%7d)" data-throw-if-not-resolved="false"></xref> is called.
After startup, <xref href="Microsoft.AI.Foundry.Local.FoundryLocalManager.Urls" data-throw-if-not-resolved="false"></xref> will contain the actual URL/s the service is listening on.
Default: 127.0.0.1:0, which binds to a random ephemeral port. Multiple URLs can be specified as a semi-colon separated list.

```csharp
public string? Urls { get; init; }
```

#### Property Value

[string](https://learn.microsoft.com/dotnet/api/system.string)?

119 changes: 119 additions & 0 deletions docs/cs-api/Microsoft.AI.Foundry.Local.Configuration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
# <a id="Microsoft_AI_Foundry_Local_Configuration"></a> Class Configuration

Namespace: [Microsoft.AI.Foundry.Local](Microsoft.AI.Foundry.Local.md)
Assembly: Microsoft.AI.Foundry.Local.dll

Foundry Local configuration used to initialize the <xref href="Microsoft.AI.Foundry.Local.FoundryLocalManager" data-throw-if-not-resolved="false"></xref> singleton.

```csharp
public class Configuration
```

#### Inheritance

[object](https://learn.microsoft.com/dotnet/api/system.object) ←
[Configuration](Microsoft.AI.Foundry.Local.Configuration.md)

#### Inherited Members

[object.Equals\(object?\)](https://learn.microsoft.com/dotnet/api/system.object.equals\#system\-object\-equals\(system\-object\)),
[object.Equals\(object?, object?\)](https://learn.microsoft.com/dotnet/api/system.object.equals\#system\-object\-equals\(system\-object\-system\-object\)),
[object.GetHashCode\(\)](https://learn.microsoft.com/dotnet/api/system.object.gethashcode),
[object.GetType\(\)](https://learn.microsoft.com/dotnet/api/system.object.gettype),
[object.MemberwiseClone\(\)](https://learn.microsoft.com/dotnet/api/system.object.memberwiseclone),
[object.ReferenceEquals\(object?, object?\)](https://learn.microsoft.com/dotnet/api/system.object.referenceequals),
[object.ToString\(\)](https://learn.microsoft.com/dotnet/api/system.object.tostring)

## Properties

### <a id="Microsoft_AI_Foundry_Local_Configuration_AdditionalSettings"></a> AdditionalSettings

Additional settings that Foundry Local Core can consume.
Keys and values are strings.

```csharp
public IDictionary<string, string>? AdditionalSettings { get; init; }
```

#### Property Value

[IDictionary](https://learn.microsoft.com/dotnet/api/system.collections.generic.idictionary\-2)<[string](https://learn.microsoft.com/dotnet/api/system.string), [string](https://learn.microsoft.com/dotnet/api/system.string)\>?

### <a id="Microsoft_AI_Foundry_Local_Configuration_AppDataDir"></a> AppDataDir

Application data directory.
Default: {home}/.{appname}, where {home} is the user's home directory and {appname} is the AppName value.

```csharp
public string? AppDataDir { get; init; }
```

#### Property Value

[string](https://learn.microsoft.com/dotnet/api/system.string)?

### <a id="Microsoft_AI_Foundry_Local_Configuration_AppName"></a> AppName

Your application name. MUST be set to a valid name.

```csharp
public required string AppName { get; set; }
```

#### Property Value

[string](https://learn.microsoft.com/dotnet/api/system.string)

### <a id="Microsoft_AI_Foundry_Local_Configuration_LogLevel"></a> LogLevel

Logging level.
Valid values are: Verbose, Debug, Information, Warning, Error, Fatal.
Default: <xref href="Microsoft.AI.Foundry.Local.LogLevel.Warning" data-throw-if-not-resolved="false"></xref>.

```csharp
public LogLevel LogLevel { get; init; }
```

#### Property Value

[LogLevel](Microsoft.AI.Foundry.Local.LogLevel.md)

### <a id="Microsoft_AI_Foundry_Local_Configuration_LogsDir"></a> LogsDir

Log directory.
Default: {appdata}/logs

```csharp
public string? LogsDir { get; init; }
```

#### Property Value

[string](https://learn.microsoft.com/dotnet/api/system.string)?

### <a id="Microsoft_AI_Foundry_Local_Configuration_ModelCacheDir"></a> ModelCacheDir

Model cache directory.
Default: {appdata}/cache/models, where {appdata} is the AppDataDir value.

```csharp
public string? ModelCacheDir { get; init; }
```

#### Property Value

[string](https://learn.microsoft.com/dotnet/api/system.string)?

### <a id="Microsoft_AI_Foundry_Local_Configuration_Web"></a> Web

Optional configuration for the built-in web service.
NOTE: This is not included in all builds.

```csharp
public Configuration.WebService? Web { get; init; }
```

#### Property Value

[Configuration](Microsoft.AI.Foundry.Local.Configuration.md).[WebService](Microsoft.AI.Foundry.Local.Configuration.WebService.md)?

Loading