Skip to content

admin_guide.md

Latest commit

 

History

History
557 lines (373 loc) · 30.2 KB

admin_guide.md

File metadata and controls

557 lines (373 loc) · 30.2 KB

Mattermost Agents Admin Guide

This guide covers installing, configuring, and managing the Mattermost Agents plugin. You'll learn how to set up AI capabilities for your Mattermost instance and configure them for your organization's needs.

Installation

Prerequisites

Before installing the Agents plugin, ensure your environment meets these requirements:

  • Mattermost Server v10.0+
  • PostgreSQL database
  • For semantic search: PostgreSQL with pgvector extension
  • Network access to your chosen LLM provider
  • If outbound LLM traffic must use an HTTP proxy, set HTTP_PROXY and HTTPS_PROXY on the Mattermost server process or container environment.
  • API keys if using a cloud LLM service

Installation Steps

Use pre-installed plugin

From Mattermost v10.3, Agents comes installed automatically and ready for you to configure a large language model (LLM). When no LLMs are configured, the Agents panel prompts users to configure one.

Install latest version

For the most recent features and improvements, you can download and install the latest plugin version from the GitHub releases page.

Install the plugin through the System Console by navigating to System Console > Plugin Management, clicking Upload Plugin, selecting the downloaded plugin file (.tar.gz), and clicking Upload. Enable the plugin after upload completes, then configure plugin settings as detailed in the Configuration section below.

Configuration

Access plugin settings

Navigate to System Console > Plugins > Agents to access the configuration interface.

Enable the plugin

Agents is enabled automatically when using the pre-installed version. If you've manually installed a newer version, you may need to enable it by going to System Console > Plugins > Agents and setting Enable Plugin to True, then complete configuration in the System Console.

Basic configuration

If you have an Enterprise, or Enterprise Advanced license, upload it to unlock additional features. If you don't have a license but are running Mattermost Enterprise Edition, an Entry license will be automatically applied for you.

For general settings, you can toggle to enable or disable the plugin system-wide, enable debug logging for troubleshooting (use only when needed), enable token usage logging for tracking LLM interactions, and configure the hostname allowlist for API calls. Outbound LLM provider traffic respects HTTP_PROXY and HTTPS_PROXY when they are set on the Mattermost server process.

AI response link rendering

Mattermost Agents includes a setting that controls whether AI-generated Markdown links are rendered as clickable links in responses:

  • System Console label: Render AI-generated links
  • Configuration key: allowUnsafeLinks
  • Default value: false

When Render AI-generated links is set to False (default), AI-generated Markdown links are shown as plain text and are not rendered as clickable links.

When this setting is set to True, AI-generated links may be rendered as clickable links. This is a security tradeoff: AI output can include malicious destinations, which can increase phishing and data exfiltration risk.

Enable this setting only in trusted or otherwise mitigated environments, such as where users are trained to validate links and your organization has endpoint protections and URL controls in place.

Service configuration

Configure an LLM provider (Service) for your Agents integration. Services manage the connection to the LLM provider, including authentication and model defaults. You can create multiple services for different providers or configurations, and share them across multiple agents.

Navigate to System Console > Plugins > Agents and select Add a Service.

Setting Description
Name Internal name for this service configuration
Type LLM provider (OpenAI, Anthropic, AWS Bedrock, Cohere, Mistral, Scale AI, Azure OpenAI, OpenAI-compatible)
API Key Your provider's API key (requirements vary by provider)
Default Model Default model to use for this service
Input Token Limit Maximum tokens allowed in input
Output Token Limit Maximum tokens allowed in output
Streaming Timeout Seconds Timeout in seconds for streaming responses
Send User ID Whether to send Mattermost user IDs to the LLM provider
Use Responses API (OpenAI/Compatible only) Enable OpenAI's Responses API for richer tool integration

Provider Specific Settings

Each provider has specific configuration requirements:

Provider Required Settings Optional Settings
OpenAI API Key Organization ID, API URL (for compatible services)
Anthropic API Key
AWS Bedrock AWS Region API Key (can use IAM role), Access/Secret Keys
Cohere API Key
Mistral API Key
Scale AI API Key, API URL Account ID (required for ScaleGov)
Azure OpenAI API Key, API URL

For AWS Bedrock, authentication can be configured using AWS credentials in the API Key/Secret fields, or by using IAM roles when running Mattermost on AWS infrastructure.

Important for Anthropic Claude models: Before using Claude models via AWS Bedrock, you must submit a one-time First Time Use (FTU) form in the AWS Bedrock Model Catalog, and attach Bedrock API permissions to your Mattermost servers' IAM role. See the AWS Bedrock setup guide for detailed instructions.

See the Provider Guide for detailed provider-specific configuration.

Agent configuration

Create an Agent (Bot) that uses a configured Service. Multiple Agents can use the same Service configuration. See license requirements for details on features that require a license.

Navigate to System Console > Plugins > Agents and select Add an Agent.

Setting Description
Display Name User-facing name shown in Mattermost
Agent Username The mattermost username for the agent. @ mentions to the agent will use this name
Agent Avatar Custom image for the agent
Service Select a configured Service from the dropdown
Model (Optional) Override the service's default model for this agent
Custom Instructions Custom instructions that define the agent's personality and capabilities
Enable Vision Enable Vision to allow the agent to process images. Requires a compatible model and service.
Enable Tools By default some tool use is enabled to allow for features such as integrations with JIRA. Disabling this allows use of models that do not support or are not very good at tool use. Some features will not work without tools.
Access Control Set which teams, channels, and users can access this agent

LLM Specific Agent Settings

Some capabilities are available depending on the selected Service and its configuration:

Setting Description
Enable Web Search Available for OpenAI (with Responses API enabled on the Service) and Anthropic. Allows the Agent to leverage the provider's native web search tool to respond with recent information.
Reasoning Enabled Available for OpenAI (with Responses API) and Anthropic. Enables "thinking" or reasoning capabilities for complex tasks.

Select Save to create the agent.

Custom instructions

Text input in the custom instructions field is included in the prompt for every request. Use this to give your agents extra context or instructions.

For example, you could list your organization's specific acronyms so the agent knows your vernacular and users can ask for definitions. Or you could give it specialized instructions like adopting a specific personality or following a certain workflow. By customizing the instructions for each individual agent, you can create a more tailored AI experience for your specific needs.

Built-in web search configuration

The built-in web search tool allows agents to retrieve current information from the internet when answering user questions. This feature is designed for deployments using LLM models that don't provide their own native web search capabilities.

When to use built-in web search

Built-in web search is intended for LLM models that lack native web search functionality. If your chosen model already provides native web search (such as OpenAI with the Responses API or Anthropic's native search tool), it's strongly recommended to use the provider's native implementation instead. Native web search tools typically offer:

  • Better integration with the model
  • More reliable search results
  • Optimized performance

For configuration details on native web search with supported providers, see the LLM Specific Settings section above.

Provider comparison

Mattermost supports two web search providers, each with varying capabilities:

Brave Search (Recommended)

Brave Search offers a superior experience for AI-powered search:

  • Purpose-built for AI: Brave's Search API is specifically designed and optimized for LLM integrations
  • Better content extraction: Returns pre-processed, LLM-ready summaries with citations
  • Fewer tool calls: Often provides complete answers without requiring follow-up web page fetches or scraping

Important: Administrators must ensure they subscribe to Brave's Pro AI plan when using this feature. Using Brave's regular Search API (non-AI tier) violates Brave's Terms of Service and may result in account suspension. The Pro AI plan is specifically licensed for AI/LLM use cases.

Google Custom Search

Google Custom Search provides access to Google's search index but has several important limitations:

  • Not a first-party integration: Mattermost uses Google's Custom Search API, which doesn't provide the same quality of results as searching directly on google.com
  • Relies on web scraping: After receiving search results, Mattermost must scrape web pages to extract content for the agent. Many websites block automated scraping or return limited content to bots
  • Rate limits: Google Custom Search has strict daily quota limits

Due to these limitations, Google Custom Search may not always provide optimal results for agent queries.

Configuration

To enable built-in web search:

  1. Navigate to System Console > Plugins > Agents > Web Search
  2. Set Enable Web Search to True
  3. Select your preferred provider from the Provider dropdown
  4. Configure provider-specific settings
Brave Search configuration
Setting Description Required
Brave API Key Your Brave Search API key (Pro AI plan) Yes
Result Limit Maximum number of results to return (1-10) No (default: 5)
API URL Override the default Brave endpoint if needed No

To obtain a Brave Search API key:

  1. Visit Brave Search API
  2. Sign up for an account
  3. Subscribe to the Pro AI plan (required for LLM usage)
  4. Generate an API key from your dashboard

Warning: Ensure you subscribe to the Pro AI plan. Using other Brave Search plans for AI/LLM integrations violates their Terms of Service.

Google Custom Search configuration
Setting Description Required
Google API Key Your Google Custom Search API key Yes
Search Engine ID Custom search engine identifier (cx parameter) Yes
Result Limit Maximum number of results to return (1-10) No (default: 5)
API URL Override the default Google endpoint if needed No

To obtain Google Custom Search credentials:

  1. Create a project in Google Cloud Console
  2. Enable the Custom Search API
  3. Create API credentials (API key)
  4. Set up a custom search engine at Google Programmable Search Engine
  5. Note the Search Engine ID (cx parameter)
Shared configuration
Setting Description
Domain Denylist Comma-separated list of domains to exclude from all search results (e.g., example.com, spam-site.org). Results from these domains are filtered out before being sent to the agent.

Usage and limitations

  • Agents are limited to 3 web searches per conversation to manage API costs and prevent LLMs from looping indefinitely
  • Agents cannot repeat the same search query within a conversation
  • Search results include clickable citations that link back to source websites
  • Domain denylisting applies to all providers and is enforced for web page fetching only.

Embed search configuration

To enable semantic search capabilities, you'll need to enable the pgvector extension in your PostgreSQL database, then configure embeddings provider settings including the provider (OpenAI, etc.), model for embeddings, and dimensions that match your chosen embedding model. Embedding search requires a license (see license requirements) and is available as an experimental feature. Performance may vary with large datasets.

Configure chunking options based on your needs:

Setting Recommended Value Description
Chunking Strategy Sentences, Paragraphs, or Fixed Size Choose based on your content type
Chunk Size 512-1024 tokens Varies by strategy
Chunk Overlap 20-50 tokens For better context continuity
Minimum Size Ratio Default Minimum ratio for chunk size validation

Run the initial indexing process after configuration.

Permission configuration

Configure who can access AI features by setting team-level, channel-level, and user-level permissions for each agent.

Management tasks

Plugin metrics

Metrics for Agents are exposed through the /plugins/mattermost-ai/metrics subpath under the existing Mattermost server metrics endpoint. This is controlled by the Listen address for performance configuration setting. It defaults to port 8067, and the following metrics are available:

  • agents_system_plugin_start_timestamp_seconds: The time the plugin started.
  • agents_system_plugin_info: The plugin version and installation ID.
  • agents_api_time_seconds: How long to execute API.
  • agents_http_requests_total: The total number of API requests.
  • agents_http_errors_total: The total number of http API errors.
  • agents_llm_requests_total: The total number of requests to upstream LLMs.

Token usage tracking

The Agents plugin can track token usage for all LLM interactions to support billing and usage analytics. When enabled, token usage data is logged to a dedicated file at logs/agents/token_usage.log in JSON format, capturing detailed information about each request:

  • User ID: The Mattermost user who initiated the request
  • Team ID: The team context for the request
  • Bot Username: Which agent was used for the interaction
  • Input Tokens: Number of tokens in the request to the LLM
  • Output Tokens: Number of tokens in the LLM response
  • Total Tokens: Combined input and output token count

To enable token usage tracking, navigate to System Console > Plugins > Agents and set Enable Token Usage Logging to True. When enabled, log files automatically rotate when they reach 100MB in size, and rotated log files are compressed to save disk space. The token usage logs provide administrators with visibility into LLM usage patterns and can be used for cost tracking and resource planning. All major LLM providers (OpenAI, Anthropic) report usage data that gets captured by this logging system.

Converting token usage logs for analysis

The token usage log file contains one JSON object per line, which is not directly compatible with tools like Microsoft Excel. Use these commands to convert the logs to different formats. Each requires jq to be installed for easy JSON parsing:

Convert to Excel-compatible JSON:

jq -s '.' logs/agents/token_usage.log > token_usage.json

Convert to CSV format:

echo "timestamp,user_id,team_id,bot_username,input_tokens,output_tokens,total_tokens" > token_usage.csv
jq -r '[.timestamp, .user_id, .team_id, .bot_username, .input_tokens, .output_tokens, .total_tokens] | @csv' logs/agents/token_usage.log >> token_usage.csv

Post indexing

Post indexing occurs automatically during initial setup and when changing embedding providers:

  1. Navigate to System Console > Plugins > Agents > Embedding Search

  2. Use the reindex controls to:

    • Monitor indexing progress during initial setup.
    • Trigger reindexing when changing embedding providers.
    • Check indexing status.

Backup and restore

The plugin configuration is stored in the Mattermost database. To backup:

  1. Ensure your regular Mattermost backup includes plugin configurations
  2. For larger deployments, consider backing up indexed vector data separately

Configuration format

The plugin uses a service-based architecture stored in the Mattermost database at PluginSettings.Plugins["mattermost-ai"]:

  • Services define LLM provider configurations (API keys, models, endpoints)
  • Bots reference services by ID and define agent personalities and access controls

This separation allows multiple bots to share the same LLM service configuration.

Configuration structure:

{
  "config": {
    "services": [
      {
        "id": "550e8400-e29b-41d4-a716-446655440000",
        "name": "OpenAI Service",
        "type": "openai",
        "apiKey": "sk-...",
        "defaultModel": "gpt-4o"
      }
    ],
    "bots": [
      {
        "id": "bot-001",
        "name": "ai",
        "displayName": "AI Assistant",
        "serviceID": "550e8400-e29b-41d4-a716-446655440000",
        "customInstructions": "You are a helpful assistant."
      }
    ]
  }
}

Supported service types: openai, anthropic, azure, openaicompatible, asage, cohere, mistral, scale

Legacy format: Older configurations with embedded service objects within bots are automatically migrated to the current format on plugin startup.

Troubleshooting

Logging

Enhanced logging can help diagnose issues:

  1. Check server logs for entries with the structured logging field plugin_ai set to mattermost-ai.
  2. Enable LLM Trace in the plugin configuration to see detailed request/response information for all LLM interactions.
  3. Enable debug logging in the plugin configuration for additional diagnostic information.
  4. For production environments, disable debug logging and LLM Trace after troubleshooting to reduce log volume.

Tool execution failures

When a tool call fails, the agent does not always stop immediately. It may continue with a follow-up model turn so it can recover, explain the failure, or answer without that tool.

To avoid endless retries, the plugin enforces a limit of three consecutive failed tool attempts. After that, no further tool calls are made for that sequence; the model is instructed to describe the latest error and ask the user for guidance or any missing information such as permissions, identifiers, or configuration details.

When users report repeated tool failures, use LLM Trace and debug logging to inspect tool errors and upstream responses. Also verify integration configuration such as API keys, endpoints, MCP connectivity, and third-party authorization, and confirm the user can access the underlying Mattermost resources the tool targets.

Integrations

Integrations are available in direct messages by default. If you enable the experimental Enable Channel Mention Tool Calling setting, @mentioning an agent in a public channel can also allow tool calling there. Native provider web search in public and private channels is controlled separately by Allow native web search in channels.

Built-in tool integrations

Server Search

  • Function: Semantic search across Mattermost content.
  • Requirements: Embedding search must be configured and enabled.
  • Security: Respects user permissions - users only see content they have access to.

User Lookup

  • Function: Look up Mattermost user information by username
  • Data Available: Username, full name, email, nickname, position, locale, timezone, last activity, status
  • Permissions: Requires VIEW_MEMBERS permission

Security Note: Tool availability and approval behavior depend on your system configuration. Tools can be configured to require explicit user approval or to run automatically under the selected tool policy.

Model Context Protocol (MCP) Integration

The Model Context Protocol (MCP) integration allows Agents to connect to external tools and services through standardized MCP servers. This feature enables expanding AI capabilities with custom integrations.

Configuration

  1. Navigate to System Console > Plugins > Agents > MCP Servers.

  2. Enable MCP integration by setting Enable MCP to True.

  3. Configure connection settings:

    • Idle Timeout: Set timeout in minutes for inactive client connections (default: 30 minutes)

Add MCP servers

  1. Select Add MCP Server to configure a new server.

  2. Configure server settings:

    • Server URL: The endpoint URL for your MCP server.
    • Custom Headers: Additional headers required by your MCP server (optional).
    • Server Name: Descriptive name for the server (auto-generated if not provided).

  3. Select Save to add the server.

Management

  • Connection Management: The system automatically manages user connections to MCP servers
  • Idle Cleanup: Inactive client connections are automatically closed after the configured timeout
  • Per-User Connections: Each user gets their own connection to MCP servers for security and isolation

Atlassian MCP server authorization

When users connect to the Atlassian MCP server, they may encounter an authorization error requiring an organization admin to authorize your Mattermost domain. This configuration must be completed in Atlassian's admin console.

To authorize your Mattermost domain:

  1. Go to admin.atlassian.com and select your organization.
  2. Go to Apps > AI settings > Rovo MCP server.
  3. Select Add domain and enter your Mattermost domain with the path wildcard: https://your-instance.mattermost.cloud/**
  4. Select Save.

Important: The /** path wildcard is required. Example domain patterns:

  • Single instance: https://your-company.mattermost.cloud/**
  • All subdomains: https://*.mattermost.cloud/**
  • Custom domain: https://chat.yourcompany.com/**

After adding the domain, wait 1-2 minutes for changes to propagate before users retry the connection.

Troubleshooting:

  • Verify you have Organization Admin permissions (Site Admin is insufficient)
  • Confirm you're configuring the organization that owns the Atlassian site
  • Ensure the domain includes https:// and the /** wildcard
  • Check for typos in the domain

For more information, see Atlassian's documentation on MCP server settings.

Note: By default, the plugin doesn't render AI-generated Markdown links (for example, JIRA ticket links) as clickable links. URLs are displayed in plain text to reduce potential phishing and data exfiltration risk. If an admin enables Render AI-generated links (allowUnsafeLinks), AI-generated links may become clickable; enable this only with appropriate trust boundaries and security mitigations in place.

Mattermost MCP Server

The Mattermost MCP Server enables AI agents and external applications to interact with your Mattermost instance through the Model Context Protocol (MCP). This is a standardized protocol that allows AI assistants to read messages, search content, create posts, and manage channels and teams programmatically.

Overview

The Mattermost MCP Server provides:

  • Direct Mattermost Integration: AI agents can access your Mattermost data and functionality through a standardized API
  • Security and Permissions: All operations respect Mattermost's permission system - users only access what they're authorized to see
  • Flexible Deployment: Available as an embedded server for Mattermost AI agents or as an HTTP server for external MCP clients
  • Rich Toolset: Comprehensive tools for reading, searching, and creating content

Use Cases

With the Mattermost MCP Server, you can:

  • Automate Channel Summaries: Ask your AI agent to summarize activity across channels, catching up on discussions while you were away.
  • Share Updates Across Channels: Have your agent post status updates to multiple channels simultaneously, keeping distributed teams synchronized.
  • Search Intelligently: Search across your entire Mattermost workspace from any MCP-enabled client to find relevant discussions, decisions, or information.
  • Coordinate Teams: Get lists of channel or team members to quickly identify who to contact or mention.
  • Automate Workflows: Use external MCP clients to automate routine tasks like posting stand-up updates, creating project channels, or notifying teams of important events.
  • Access Context-Aware Assistance: AI agents can read conversation threads to understand context before responding or taking action.

Available Tools

The MCP server provides the following tools to AI agents and external clients:

  • read_post: Read a specific post and its thread
  • read_channel: Retrieve recent posts from a channel
  • search_posts: Search across Mattermost content with optional team/channel filters
  • create_post: Create new posts or replies in channels
  • create_channel: Create new public or private channels
  • get_channel_info: Retrieve channel details by ID or name
  • get_team_info: Retrieve team details by ID or name
  • search_users: Find users by username, email, or name
  • get_channel_members: List all members of a channel
  • get_team_members: List all members of a team

Deployment

MCP Server Configuration

For AI Agents

To set up an embedded MCP server providing Mattermost AI agents with direct access to Mattermost functionality:

  1. Go to System Console > Plugins > Agents > MCP Servers.
  2. Set Enable Embedded Server to True.
  3. When enabled, all configured agents can access Mattermost tools.

Agents will automatically use these tools when appropriate to complete user requests.

For External Clients

You can enable external MCP clients, such as Claude web, Claude Code, or other MCP-compatible applications, to interact with your Mattermost instance.

Requirements:

  • Mattermost Server v11.2 or later
  • Valid authentication method (OAuth or Personal Access Token)

Note: The server uses streamable HTTP transport and does not support traditional Server-Sent Events (SSE) transport. External clients must use the streamable HTTP transport available at the /mcp endpoint.

To enable an external MCP client:

  1. Go to System Console > Plugins > Agents > MCP Servers
  2. Set Enable Mattermost MCP Server (HTTP) to True. The MCP server will be available at: https://your-mattermost-server/plugins/mattermost-ai/mcp-server/mcp

Authentication:

OAuth 2.0

The MCP server supports OAuth 2.0 authentication with both manual and automatic client registration.

Prerequisites:

  • Enable OAuth 2.0 service provider in System Console > Integrations > Integration Management:
    • Set Enable OAuth 2.0 Service Provider to True
    • For automatic client registration, set Enable OAuth 2.0 Dynamic Client Registration to True (Note: DCR is an unprotected endpoint, meaning it is publicly accessible and does not require authentication—anyone can register OAuth clients if this feature is enabled. See the OAuth 2.0 documentation for security considerations.)

Client Registration Methods:

  • Dynamic Client Registration (DCR/RFC 7591): External clients can automatically register and obtain credentials without manual setup.
  • Manual Registration: Create OAuth applications through Product menu > Integrations > OAuth 2.0 Applications. See the OAuth 2.0 documentation for details.

Additional Details:

  • Supports both public clients (e.g., desktop applications) and confidential clients (e.g., server applications)
  • Authorization through standard Mattermost OAuth flows
  • OAuth metadata endpoints:
    • Protected resource metadata: https://your-mattermost-server/plugins/mattermost-ai/mcp-server/.well-known/oauth-protected-resource
    • Authorization server metadata: https://your-mattermost-server/.well-known/oauth-authorization-server

Personal Access Tokens

You can authenticate using Mattermost Personal Access Tokens (PAT):

  1. Create a Personal Access Token in Mattermost (User Settings > Security > Personal Access Tokens).
  2. Configure your MCP client to use Bearer token authentication with the PAT.

License requirements

The following table outlines which features require a license:

Feature License Required
Basic agent configuration (single agent) No license required
Chat with agents in DMs and channels No license required
Image analysis (vision capabilities) No license required
Basic tool integrations No license required
Multiple agent configurations Entry, Enterprise, and Enterprise Advanced
Fine-grained access controls Entry, Enterprise, and Enterprise Advanced
Embedding search (semantic AI search) Entry, Enterprise, and Enterprise Advanced
MCP Support Entry, Enterprise, and Enterprise Advanced
Usage analytics and token tracking Entry, Enterprise, and Enterprise Advanced
AI Actions menu (thread summarization) Entry, Enterprise, and Enterprise Advanced
Channel summarization (unread messages) Entry, Enterprise, and Enterprise Advanced
Recorded meeting transcripts and summarization Entry, Enterprise, and Enterprise Advanced