feat: add pyproject.toml and run uv sync for all the plugins

[Stream29]

Related Issues or Context

Fixes #2253 .

This PR contains Changes to Non-Plugin

• Documentation
• Other

This PR contains Changes to Non-LLM Models Plugin

• I have Run Comprehensive Tests Relevant to My Changes

This PR contains Changes to LLM Models Plugin

• My Changes Affect Message Flow Handling (System Messages and User→Assistant Turn-Taking)

• My Changes Affect Tool Interaction Flow (Multi-Round Usage and Output Handling, for both Agent App and Agent Node)

• My Changes Affect Multimodal Input Handling (Images, PDFs, Audio, Video, etc.)

• My Changes Affect Multimodal Output Generation (Images, Audio, Video, etc.)

• My Changes Affect Structured Output Format (JSON, XML, etc.)

• My Changes Affect Token Consumption Metrics

• My Changes Affect Other LLM Functionalities (Reasoning Process, Grounding, Prompt Caching, etc.)

• Other Changes (Add New Models, Fix Model Parameters etc.)

Version Control (Any Changes to the Plugin Will Require Bumping the Version)

• I have Bumped Up the Version in Manifest.yaml (Top-Level Version Field, Not in Meta Section)

Dify Plugin SDK Version

• I have Ensured dify_plugin>=0.3.0,<0.6.0 is in requirements.txt (SDK docs)

Environment Verification (If Any Code Changes)

Local Deployment Environment

• Dify Version is: , I have Tested My Changes on Local Deployment Dify with a Clean Environment That Matches the Production Configuration.

SaaS Environment

• I have Tested My Changes on cloud.dify.ai with a Clean Environment That Matches the Production Configuration

[gemini-code-assist]

Summary of Changes

Hello @Stream29, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the project's dependency management by introducing pyproject.toml files for every plugin, datasource, model, and tool. This strategic move aims to standardize how dependencies are declared and managed across the entire repository, enabling the adoption of modern tools like uv for more efficient and reproducible builds. The change ensures that each component clearly defines its requirements, promoting better maintainability and simplifying future development and deployment workflows.

Highlights

• Standardized Dependency Management: Introduced pyproject.toml files across all Dify plugins, datasources, models, and tools to standardize project metadata and dependency declarations.
• uv Integration: The addition of pyproject.toml files paves the way for using uv (a fast Python package installer and resolver) for consistent and efficient dependency management, as indicated by the commented uv pip compile command in each file.
• Python Version Requirement: Each pyproject.toml specifies a minimum Python version of >=3.12, ensuring compatibility with modern Python environments.
• Component-Specific Dependencies: Individual pyproject.toml files define specific dependencies for each component (e.g., dify_plugin, boto3, openai), allowing for granular control over package versions.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request is a great step towards modernizing the dependency management for all plugins by introducing pyproject.toml files. The changes are extensive and set a good foundation. I've reviewed the new files and have identified a few common areas for improvement to enhance consistency and maintainability across all plugins. I've added specific comments on a few files as examples, but the feedback applies to most of the new pyproject.toml files.

Key areas for improvement:

• Placeholder Descriptions: Many plugins have a placeholder description ("Add your description here"). These should be updated with a concise and meaningful description of what each plugin does.
• Package Naming Consistency: There's an inconsistency in the naming of the dify-plugin package. It's sometimes referred to as dify_plugin and sometimes dify-plugin. The canonical name on PyPI is dify-plugin, and it's best practice to use that consistently.
• Development Dependencies: Development tools like black and ruff are mentioned in comments but not listed as dependencies. They should be moved into a [project.optional-dependencies] group (e.g., dev) for a more robust development setup.
• Dependency Pinning: Some dependencies are not pinned (e.g., boto3). This can lead to unexpected issues when new versions are released. It's recommended to pin dependencies using ~= or == to ensure reproducible builds.
• Missing Metadata: Some pyproject.toml files are missing the readme field, which is good to have for package metadata.
• Transitive Dependencies: One file (tools/bitbucket/pyproject.toml) incorrectly lists transitive dependencies. Only direct dependencies should be listed.

Please apply these suggestions across all the new pyproject.toml files for consistency. Great work on this large-scale refactoring!

[Yeuoly]

Did you have some tests to ensure that requirements.txt generated by those pyproject.toml were not changed compare to the original files on the repo?

Since it's a huge change, we must to enforce that no breaking changes was included.

A) What's more, I intent to rewrite those requirements.txt and use CI/CD pipeline to update it, let's take a example:
Whenever a pyproject.toml was changed in a commit, CI/CD should detect the change and regenerate a requirements.txt to replace the original one, unless there isn't any difference between the two files.

But the proposal A could be done in another PR, not this one, it will lead the PR to have a too large scope.

[Yeuoly]

BTW, I think we need also update the dify-plugin commandline there https://github.com/langgenius/dify-plugin-daemon to introduce pyproject.toml

[Stream29]

BTW, I think we need also update the dify-plugin commandline there https://github.com/langgenius/dify-plugin-daemon to introduce pyproject.toml

@Yeuoly I'm kind of confused.

Will the final .difypkg includes pyproject.toml and uv.lock? Will plugin-daemon try to use pyproject.toml and uv.lock to install dependencies to venv?

If yes, this may be a breaking change and require document update.
If no, pyproject.toml and uv.lock is only a development convenience. Then this pull request can be merged already.

[Stream29]

@Yeuoly I've checked the consistency. Except for some non-existing dependencies (eg, requests>=2.32.7 in tools\gmail), all the requirements.txt exported by uv pip compile contains all the dependencies from requirements.txt.

This PR is safe to merge.

I'll create another PR to fix the non-existing dependencies.

[kurokobo]

@Yeuoly @Stream29
Thanks for working on this, I'm just a contributor so this is my personal opinion.
I'm in favor of introducing pyproject.toml and making uv-based development the mainstream approach.

However, in my opinion, this PR is a bit too hasty. Without updates to the documentation and consistency with the Plugin Daemon implementation, developers might get confused in the following ways:

• Do developers always have to maintain both pyproject.toml and uv.lock?
• Do developers still need to maintain requirements.txt?
• If pyproject.toml and requirements.txt aren't kept in sync, how does the plugin behave?
• Are pyproject.toml and uv.lock actually used by the Plugin Daemon?

In reality, if this PR is merged, the difypkg files of official plugins will start to include both pyproject.toml and uv.lock, and requirements.txt if exist (and we’ll need to update .difyignore if you want to exclude them).
However, the Plugin Daemon currently installs dependencies using the uv pip install -r requirements.txt command, so it only needs requirements.txt—it ignores uv.lock.

This behavior isn’t very intuitive and lacks consistency, which can be confusing for developers. Personally, if we’re going to move to uv-based development, I’d prefer the following steps:

• Update the Plugin Daemon so that, when installing plugins, it prioritizes pyproject.toml and uv.lock if they exist, and falls back to requirements.txt otherwise.
• Change the plugin init command to generate pyproject.toml instead of requirements.txt.
• Update the developer documentation to clearly state the recommended maintenance practices for pyproject.toml and uv.lock. For example, is it acceptable if the version in manifest.yaml and pyproject.toml don’t match? What is the recommended action if both pyproject.toml and requirements.txt exist?
• If needed, update this repo’s and dify-plugins’s GHA to validate pyproject.toml and uv.lock, and requirements.txt.
• Finally, remove requirements.txt from existing plugins and use only pyproject.toml and uv.lock instead.

How do you think?

[Stream29]

I agree with most of your opinions. @kurokobo

• Plugin daemon will prioritize pyproject.toml and uv.lock if they exist.
• plugin init command will generate pyproject.toml instead of requirements.txt.
• Developer documentation will be updated to clearly state the recommended maintenance practices.
• CI will be introduced on this repo, validating pyproject.toml and executing tests based on it.
• For official plugins, we will move to pyproject.toml and uv.lock.
• Whether to remove requirements.txt for official plugins should further discussed.

[Yeuoly]

@Yeuoly I've checked the consistency. Except for some non-existing dependencies (eg, requests>=2.32.7 in tools\gmail), all the requirements.txt exported by uv pip compile contains all the dependencies from requirements.txt.

This PR is safe to merge.

I'll create another PR to fix the non-existing dependencies.

dify-plugin itself has integrated with requests already.

[Stream29]

validate_requirements.py

I used this script to validate all the pyproject.toml can satisfy requirements.txt.

The output is as following:

Output.txt