Fix codeql alert + modularize pytorch codegen #49
+6,587
−2,592
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add LayerCodeSpec dataclass for structured code specifications - Create TemplateManager for Jinja2 template loading and caching - Add get_pytorch_code_spec() and get_tensorflow_code_spec() methods to NodeDefinition - Create template directory structure for PyTorch and TensorFlow - Add base layer templates for both frameworks
- Create layer templates for conv2d, linear, maxpool, flatten, relu, softmax, dropout, batchnorm, attention, add, concat - Create file templates for model.py, train.py, dataset.py, config.py - Templates preserve good coding practices: reusable classes, comprehensive documentation, proper shape annotations
- Add base utilities (topological sort, edge map builder) - Create PyTorchCodeOrchestrator class - Implement template-driven code generation - Support skip connections in forward pass - Preserve all existing features (adaptive hyperparameters, test code generation) - Return same API format as original codegen
- Modify generate_pytorch_code() to delegate to PyTorchCodeOrchestrator - Preserve exact same function signature and return type - Maintain 100% backward compatibility with views - Mark legacy code as preserved for reference - Complete refactor: template-based, extensible, non-monolithic
- Create layer templates for conv2d, linear, maxpool, flatten, dropout, batchnorm, add, concat - Create file templates for model.py, train.py, dataset.py, config.py - Adapt templates for TensorFlow/Keras API and NHWC format - Maintain same documentation quality as PyTorch
- Add get_tensorflow_code_spec() to Conv2D, Linear, MaxPool2D, Flatten, Dropout, BatchNorm2D, Add, Concat - Adapt for TensorFlow/Keras API (filters vs channels, NHWC format) - Mirror PyTorch implementation pattern for consistency
- Add TensorFlowCodeOrchestrator class mirroring PyTorch structure - Implement template-driven code generation for TensorFlow/Keras - Support NHWC format and TensorFlow-specific patterns - Handle skip connections in forward pass - Return same API format as original TensorFlow codegen
- Modify generate_tensorflow_code() to delegate to TensorFlowCodeOrchestrator - Preserve exact same function signature and return type - Maintain 100% backward compatibility with views - Mark legacy code as preserved for reference - Complete TensorFlow refactor: template-based, extensible, non-monolithic
- Fix: Remove explicit .forward() and .call() method calls (use __call__ instead) - Fix: PyTorch layers should be called directly, not via .forward() - Fix: TensorFlow layers should be called directly, not via .call() - Add: Base orchestrator class to eliminate WET code (DRY principle) - Prepare: Foundation for refactoring both orchestrators to inherit from base
- Fix: Change import from .pytorch_codegen to .enhanced_pytorch_codegen - The actual file is named enhanced_pytorch_codegen.py, not pytorch_codegen.py
- Remove imports of classes that don't exist (GroupBlockShapeComputer, safe_get_shape_data, etc.) - These were only used in legacy code that never executes due to early delegation - Fixes ModuleNotFoundError on server startup
…age and enhance edge mapping for skip connections
security fixes + codegen improvement
…-ZROiF Claude/refactor pytorch codegen zr oi f
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces a new code generation orchestration system for neural network model code, supporting both PyTorch and TensorFlow backends. It adds a modular, extensible architecture for generating code from graph-based model definitions, including support for group blocks (nested reusable blocks), and provides a shared base for framework-specific code generation. The changes include new orchestrator base classes, utility functions for graph processing, and a group block generator abstraction. Additionally, a CodeQL workflow is added for security scanning.
Core code generation architecture:
BaseCodeOrchestratorabstract class (base_orchestrator.py) to encapsulate shared orchestration logic for code generation, including topological sorting, edge mapping, code spec generation, and rendering for both PyTorch and TensorFlow backends. This class provides extensible hooks for framework-specific implementations.base.pywith shared utility functions for topological sorting, input variable mapping, and node type/config extraction, to be used by orchestrators and code generators.__init__.pyin the codegen package to expose the orchestrators for PyTorch and TensorFlow, establishing a clear import interface.Group block (nested block) support:
GroupBlockGeneratorabstract base class (group_block_generator.py) to handle code generation for group (composite) blocks, including parsing port mappings, sorting internal nodes, detecting nested groups, and preparing template contexts for rendering. This enables reusable, hierarchical model components across frameworks.DevOps and security:
.github/workflows/codeql.yml) to enable advanced static analysis and security scanning for Python and JavaScript/TypeScript code in the repository.