feat: generic redesign with Store[T], typed builders, and dual type params#16
Merged
feat: generic redesign with Store[T], typed builders, and dual type params#16
Conversation
Full architecture plan to make go-wf fully generic with compile-time type safety across workflow core, container, function, and store packages.
23-task phased plan: store → core types → container → function → datasync → cleanup. Each phase compiles and tests independently.
Port S3 storage from workflow/artifacts/s3.go to the new RawStore interface with simplified key-based API and no metadata dependency. Includes integration test using testcontainers with rustfs/MinIO.
Wrap RawStore with OpenTelemetry instrumentation following the same pattern as workflow/artifacts/otel.go. Zero overhead when OTel config is not present in context.
…types Add phantom type parameter O TaskOutput to PipelineInput, ParallelInput, LoopInput, and ParameterizedLoopInput for compile-time I/O type pairing.
Add generic DAG types (DAGNode, DAGInput, DAGOutput, NodeResult) to the workflow core package with DFS-based cycle detection validation ported from function/payload. These are type definitions only — no execution logic, keeping container/function concerns separate.
Add function/types.go and function/types_extended.go with type aliases pointing to function/payload, allowing callers to use function.FunctionExecutionInput directly. Update worker.go to use the local aliases. Mark Pipeline/Parallel input/output types in payload as deprecated in favor of generic workflow types.
… access Re-export ContainerExecutionInput, ContainerExecutionOutput, WaitStrategyConfig, and all extended types (DAGNode, DAGWorkflowInput, etc.) from the container package root via type aliases pointing to container/payload. This allows consumers to use container.ContainerExecutionInput instead of the longer container/payload.ContainerExecutionInput path.
Make WorkflowBuilder and LoopBuilder generic with [I TaskInput, O TaskOutput] type parameters. Add NewFunctionBuilder, NewFunctionLoopBuilder convenience constructors. Update patterns to use generic types (Tasks instead of Functions). Update all tests and examples accordingly.
… builder Add GenericBuilder[I, O] type that works with any TaskInput/TaskOutput types. Keep existing WorkflowBuilder for backward compatibility with container-specific workflows. Add BuildGenericPipeline and BuildGenericParallel methods that produce workflow.PipelineInput and workflow.ParallelInput generic types.
…nd parallel Add GenericContainerPipelineWorkflow and GenericParallelContainersWorkflow that accept generic workflow.PipelineInput/ParallelInput types directly, eliminating the need for type conversion. Keep legacy wrappers for backward compatibility.
Add SubmitTypedWorkflow and SubmitAndWaitTyped generic functions that provide
compile-time type safety for workflow submission, eliminating the need for
interface{} type switches. Keep legacy SubmitWorkflow and SubmitAndWait for
backward compatibility.
…ates Add GenericSource[I] interface, GenericSourceFunc[I] adapter, and TaskInputSource[I] wrapper alongside the existing container-specific WorkflowSource interface. This enables templates and sources to work with any TaskInput type in the generic builder.
…irectly Workflow functions now accept generic workflow.PipelineInput[I,O], ParallelInput[I,O], LoopInput[I,O] directly instead of function-specific payload types. Delete helpers.go since conversion functions are no longer needed. Update all workflow tests accordingly.
…types Update integration tests to use workflow.PipelineInput[I,O], ParallelInput[I,O], LoopInput[I,O], and ParameterizedLoopInput[I,O] directly instead of the deprecated payload-specific types. Mark Pipeline/Parallel input/output types in payload as deprecated.
…wStore Replace workflow/artifacts dependency with workflow/store in datasync package. Job.ArtifactConfig becomes Job.Store (store.RawStore) and builder.WithArtifactConfig becomes builder.WithStore.
- Add statusFailed constant in container/operations.go (goconst) - Remove unused parallelMode field from GenericBuilder (unused) - Fix gofumpt formatting for long function signatures - Add nolint directives for intentional error ignoring (errcheck) - Update INSTRUCTION.md: add workflow/store/ to key paths, document generic store, dual type parameters, and generic builder alternatives
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
workflow/store/package — genericStore[T],RawStore,Codec[T],TypedStore[T]replacing the metadata-coupledworkflow/artifacts/for new code. IncludesLocalStore,S3Store,InstrumentedStore(OTel),KeyBuilder.PipelineInput[I, O],ParallelInput[I, O],LoopInput[I, O]now carry both input and output types for compile-time I/O pairing.DAGNode[I, O],DAGInput[I, O],DAGOutput[O]inworkflow/with cycle detection validation.GenericBuilder[I, O], generic workflow entry points, typed operations (SubmitTypedWorkflow[I],SubmitAndWaitTyped[O]), genericWorkflowSource[I]templates. Backward compatible — existing APIs preserved alongside new generic ones.WorkflowBuilder[I, O]andLoopBuilder[I, O]fully generic, workflow wrappers simplified (helpers.go deleted), patterns updated.Job[T, U].ArtifactConfig→Job[T, U].Store store.RawStore.Test plan
task test:unit(zero failures)task lintpassestask fmtproduces no changestask test:integration)