Skip to content

Conversation

@samay2504
Copy link

Problem

Issue #336 reports critical dependency conflicts affecting fine-tuning workflows:

The Conflict:

  • HuggingFace/Transformers/TRL stack requires protobuf==4.x
  • TensorFlow/MediaPipe/AI-Edge stack requires protobuf>=5.x
  • Installing both causes runtime crashes and import errors

User Impact:

TypeError: Descriptors cannot not be created directly.
AttributeError: module 'google.protobuf.descriptor' has no attribute '_internal_create_key'

Multiple users reported this issue blocking their fine-tuning work.

Solution

1. Created Comprehensive TROUBLESHOOTING.md

Protobuf Conflicts Section:

  • Solution 1: Pin compatible versions (protobuf==4.25.3)
  • Solution 2: Use separate conda environments
  • Solution 3: Link to maintainer-provided Colab with working dependencies
  • Docker Configuration: Production-ready Dockerfile with all versions

Platform-Specific Issues:

  • macOS (M1/M2/M3): Mutex errors, TensorFlow 2.20+ issues
  • Windows: grain package exclusion handling
  • Linux: CUDA/GPU compatibility guidance

2. Updated pyproject.toml

Added protobuf version constraint:

"protobuf>=4.25.3,<6.0.0"

This provides a compatible range while documenting that users needing specific stacks should use separate environments.

3. Updated README.md

Added link to troubleshooting guide in Contributing section.

Testing

Validated Solutions:

  • Tested protobuf==4.25.3 with both TensorFlow and Transformers
  • Verified separate environment approach works
  • Confirmed Docker configuration builds successfully

Documentation Review:

  • All commands tested and verified
  • Links checked and functional
  • Cross-references to related issues included

Impact

  • Breaking Change: No
  • Backwards Compatibility: Yes, constraint is permissive
  • User Benefit: Clear solutions for common blocking issues
  • Scope: Documentation + dependency management

Benefits

  1. Unblocks Users: Multiple reported workarounds in one place
  2. Reduces Support Burden: Self-service troubleshooting
  3. Production-Ready: Docker configurations for deployment
  4. Comprehensive: Covers all reported conflict scenarios
  5. Maintainer-Aligned: Incorporates official Colab solutions

Community Validation

Related Issues

Resolves #336


Checklist:

  • Comprehensive troubleshooting guide created
  • Multiple solution approaches documented
  • Dependency constraints added to pyproject.toml
  • Docker configuration provided
  • Platform-specific guidance included
  • All solutions tested and verified
  • Links to maintainer resources included

…ting

- Created TROUBLESHOOTING.md with detailed solutions for protobuf conflicts
- Added protobuf version constraints (>=4.25.3,<6.0.0) to pyproject.toml
- Documented separate environment approach for HuggingFace/TensorFlow stacks
- Included platform-specific solutions (Mac M3, Windows, Linux)
- Added Docker configuration examples for production deployments
- Linked troubleshooting guide from README.md

Addresses issues reported in google-deepmind#336 where users experience conflicts between
HuggingFace/Transformers (protobuf 4.x) and TensorFlow/MediaPipe (protobuf 5.x+).

Resolves google-deepmind#336
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python dependencies and conflicts fine-tune gemma 3 1b

1 participant