API Compatibility Issue: "Invalid schema for response_format 'CustomerAction'" when running simulations

[gregjagiello]

Issue Description

Hi Magentic Marketplace team...Greg here, I'm a not-very-technical product designer and product manager, trying to become way more hands-on familiar with agentic workflows. I'm using Manus as my helper and everhthing written below is directly from my chats with Manus:

I'm a newcomer to the project and excited to learn about agentic marketplaces. I successfully set up the development environment following the README instructions, but I'm encountering an error when trying to run my first simulation.

What I'm Trying to Do

Run the sample Mexican restaurant marketplace experiment:

magentic-marketplace run data/mexican_3_9 --experiment-name test

What's Happening

The simulation starts but fails with this error:

Error code: 400 - {'error': {'type': 'invalid_request_error', 
'message': "API returned unexpected status code: 400: Invalid schema for 
response_format 'CustomerAction': In context=('properties', 'messages'), 
schema must have a 'type' key.", 'code': '11110'}}

The error repeats for each customer agent and the simulation cannot complete.

My Setup

• Repository: Latest from main branch (commit bb8d504)
• Python: 3.11.0rc1
• Installation: Used uv sync --all-extras as documented
• Database: PostgreSQL running via Docker
• Models Tested:
  • gpt-4.1-nano
  • gpt-4.1-mini
• API: OpenAI-compatible endpoint

Environment Configuration

export LLM_PROVIDER="openai"
export LLM_MODEL="gpt-4.1-mini"
export LLM_REASONING_EFFORT="minimal"
export LLM_MAX_CONCURRENCY="64"

Error Location

The error occurs in:

packages/magentic-marketplace/src/magentic_marketplace/marketplace/agents/customer/agent.py
Line 179, in _generate_customer_action

Questions

1. Which specific OpenAI models have been tested and confirmed working? The README doesn't specify model compatibility.
2. Are there known issues with OpenAI-compatible API endpoints? I'm not using the official OpenAI API.
3. Is there a workaround or configuration I'm missing?

Why I'm Reporting This

I'm really interested in contributing to this project, especially in designing new marketplace scenarios (I have a healthcare scheduling use case in mind!). However, I can't test my ideas without being able to run simulations.

I think this issue might affect other newcomers trying to get started with the framework. It would be helpful to have:

• Documentation about which models/APIs are compatible
• Clearer error messages or validation before API calls
• A troubleshooting guide for common setup issues

What I Can Help With

Even though I'm not a hardcore developer, I'd be happy to:

• Test fixes once they're available
• Help document the solution for other newcomers
• Contribute scenario designs once I can run simulations
• Improve the getting-started documentation based on my experience

Additional Context

I'm approaching this as someone interested in agentic AI collaboration but without deep technical expertise. The setup process was smooth until hitting this API compatibility issue. Any guidance would be greatly appreciated!

Thank you for creating this amazing open-source project! 🙏

---

Labels to Add (if you have permission)

• bug
• help wanted
• documentation
• good first issue

[willeppy]

Hey @gregjagiello thanks for bringing this up! So I wasn't able to reproduce but here's some ideas about what might be going on---

This error comes up when a model is not producing a stuctured output correctly. We have some built in retries but the simulation might crash if it fails too many times. However, I'm able to run successfully with smaller models like gpt-4.1-mini and gpt-4.1-nano but they definitley take more cycles.

Can you run a simulation successfully with a newer/bigger model like just gpt-4.1 or gpt-5? In our experiments these pretty reliably produce the correct stuctured outputs.

[willeppy]

Also how are you making model calls? You said you're not using the offical openai endpoint? That might be a source of error

[gregjagiello]

Thank you for the reply @willeppy , I'm digging back in soon and will use your feedback and report back.