Custom tools do not work with binary agent server builds

[xingyaoww]

Problem

When the agent server is bundled into a binary executable using PyInstaller (the binary target in the Dockerfile), custom tools cannot be dynamically imported at runtime.

Root Cause

The current implementation in PR #1383 uses importlib.import_module() to dynamically load custom tool modules:

if request.tool_module_qualnames:
    import importlib
    for tool_name, module_qualname in request.tool_module_qualnames.items():
        importlib.import_module(module_qualname)

This works in source mode where a real Python interpreter is available, but fails in binary mode because:

1. PyInstaller creates a frozen executable with all Python modules bundled inside
2. The frozen Python interpreter can only import modules that were bundled at build time
3. External Python files copied into the Docker image are not accessible to the binary's internal importer
4. importlib.import_module() will raise ModuleNotFoundError for any module not bundled in the binary

Context

This was identified during review of PR #1383 (feat: Add support for custom tools with remote agent server). The PR works correctly for source-mode deployments but not for binary-mode deployments.

See: #1383 (comment)

Potential Solutions

1. Use Source Mode for Custom Tools (Simplest)

• The Dockerfile already has source and source-minimal targets
• Users needing custom tools should use source mode instead of binary mode
• Pros: Works with existing implementation, no code changes needed
• Cons: Larger image size, slower startup

2. HTTP-based Tool Execution (Most Flexible)

• Custom tools run as separate HTTP services
• Agent server calls tools via HTTP instead of importing them
• Similar to MCP (Model Context Protocol) approach
• Pros: Language-agnostic, works with binary, clean separation
• Cons: More complex setup, network overhead

3. Build Custom Binary with Tools Included

• Modify the build process to include custom tools at build time
• Users provide their tools, and a custom binary is built that includes them
• Pros: Single binary with everything included
• Cons: Requires rebuild for each tool set

4. Dynamic Code Execution

• Send tool code as a string instead of module qualnames
• Server executes the code using exec() to define the tool class
• Pros: Works with binary, no filesystem needed
• Cons: Security concerns, harder to debug

Recommendation

For now, document that custom tools require source mode deployment. The binary mode should be used for standard deployments without custom tools.

Longer term, consider implementing HTTP-based tool execution for maximum flexibility.

Related

• PR feat: Add support for custom tools with remote agent server #1383: feat: Add support for custom tools with remote agent server
• Issue Document and implement custom tool support for remote agent server #1381: Support custom tools with remote agent server

[xingyaoww]

@OpenHands please review server.yml that current build our binary image, do you think if it is possible to add a custom tool on top of a binary executable? If not, do you think if it make sense to add a source variant for all the server images we are currently building?

[openhands-ai]

I'm on it! xingyaoww can track my progress at all-hands.dev

[openhands-ai]

Summary

I've reviewed the server.yml workflow and addressed your questions:

Question 1: Is it possible to add a custom tool on top of a binary executable?

No, it is not possible. PyInstaller creates a frozen executable with all Python modules bundled inside. The frozen Python interpreter can only import modules that were bundled at build time. External Python files copied into the Docker image are not accessible to the binary's internal importer, so importlib.import_module() will raise ModuleNotFoundError for any module not bundled in the binary.

Question 2: Does it make sense to add a source variant for all the server images?

Yes, it makes sense. I've implemented this solution by modifying the server.yml workflow to build both binary and source variants for all server images.

Changes Made

I created PR #1536 (#1536) with the following changes to .github/workflows/server.yml:

1. Expanded build matrix: From 6 jobs to 12 jobs (3 variants × 2 targets × 2 architectures)

   • Binary target: Production mode with PyInstaller binary (smaller, faster startup)
   • Source target: Development mode with Python source (supports custom tools)

2. Updated merge-manifests job: Now handles both binary and source variants with proper tag suffixes

3. Updated consolidate-build-info: Tracks target information for PR descriptions

4. Updated PR description template: Documents binary vs source targets with clear guidance

New Image Tags

After this change, users can choose between:

• ghcr.io/openhands/agent-server:<sha>-python - Binary build (production)
• ghcr.io/openhands/agent-server:<sha>-python-source - Source build (supports custom tools)

This closes issue #1531 by providing source mode images that support custom tools via dynamic module loading, while maintaining binary mode images for production deployments.

_{View full conversation}

[all-hands-bot]

[Automatic Post]: This issue has been waiting for triage. @raymyers, could you please take a look and add the appropriate priority label when you have a chance?

[all-hands-bot]

[Automatic Post]: This issue has been waiting for triage. @raymyers, could you please take a look and add the appropriate priority label when you have a chance?

[enyst]

Out of curiosity, why haven’t we chosen MCP? I’m not suggesting it, I’m just not sure I understand the need

[xingyaoww]

@enyst this PR mainly refers to the need of "passing custom tools" to local SDK, and then try to get these custom tools to work on remote agent server, if that make sense?

I think start MCP server could involves a bit more complication and less straightforward than our existing solution.

OTOH, user can always define their own server and connect to it via configuration

[all-hands-bot]

[Automatic Post]: This issue has been waiting for triage. @raymyers, could you please take a look and add the appropriate priority label when you have a chance?

[all-hands-bot]

[Automatic Post]: This issue has been waiting for triage. @raymyers, could you please take a look and add the appropriate priority label when you have a chance?

[all-hands-bot]

[Automatic Post]: This issue has been waiting for triage. @csmith49, could you please take a look and add the appropriate priority label when you have a chance?