Development Workflow Guide for Control Toolbox

[ocots]

Development Workflow Guide for Control Toolbox

Overview

This guide explains how to efficiently develop across multiple Control Toolbox packages using Julia's Pkg.develop() feature. This workflow is designed for small teams (3-4 developers) working on interdependent packages.

Key Principle: Develop all packages locally simultaneously, ignoring version constraints during development. Register versions only when features are complete and tested.

---

One-Time Setup

1. Clone All Repositories

Create a dedicated folder and clone all repositories side-by-side:

mkdir -p ~/dev/control-toolbox
cd ~/dev/control-toolbox

git clone https://github.com/control-toolbox/CTBase.jl.git
git clone https://github.com/control-toolbox/CTModels.jl.git
git clone https://github.com/control-toolbox/CTParser.jl.git
git clone https://github.com/control-toolbox/CTDirect.jl.git
git clone https://github.com/control-toolbox/CTFlows.jl.git
git clone https://github.com/control-toolbox/OptimalControl.jl.git

Result:

~/dev/control-toolbox/
├── CTBase.jl/
├── CTModels.jl/
├── CTParser.jl/
├── CTDirect.jl/
├── CTFlows.jl/
└── OptimalControl.jl/

2. Set Up Development Environment

Navigate to your main working package (typically OptimalControl) and activate development mode for all dependencies:

cd OptimalControl.jl
julia --project

In Julia REPL:

using Pkg

# Develop all local packages
Pkg.develop(path="../CTBase.jl")
Pkg.develop(path="../CTModels.jl")
Pkg.develop(path="../CTParser.jl")
Pkg.develop(path="../CTDirect.jl")
Pkg.develop(path="../CTFlows.jl")

# Verify setup
Pkg.status()

You should see (dev) markers:

Status `~/dev/control-toolbox/OptimalControl.jl/Project.toml`
  [e2007a6d] CTBase v0.16.0 `../CTBase.jl` (dev)
  [9f9ead1c] CTModels v0.10.2 `../CTModels.jl` (dev)
  ...

What this means: All packages now use your local code, completely ignoring version constraints.

3. Create a Setup Script (Optional but Recommended)

Create ~/dev/control-toolbox/dev-setup.jl:

using Pkg

packages = [
    "CTBase.jl",
    "CTModels.jl",
    "CTParser.jl",
    "CTDirect.jl",
    "CTFlows.jl"
]

println("🔧 Setting up development environment...")
for pkg in packages
    println("  → Developing: $pkg")
    Pkg.develop(path="../$pkg")
end
println("✅ Development environment ready!")

Usage in any project:

julia> include("../dev-setup.jl")

---

Daily Development Workflow

Working on a Single Package

Example: Improving CTBase

cd ~/dev/control-toolbox/CTBase.jl
git checkout -b feature/improve-solver

# Edit code...
# CTBase.jl/src/solver.jl

Test in isolation:

julia --project=.
using Pkg
Pkg.test()

Test with dependents:

cd ../OptimalControl.jl
julia --project=.
using Pkg
Pkg.test()  # Uses your local CTBase automatically!

Key advantage: Changes in CTBase are immediately available to all dependent packages. No registration, no waiting.

Working Across Multiple Packages

Example: API change in CTBase affecting CTModels

# 1. Change CTBase
cd ~/dev/control-toolbox/CTBase.jl
git checkout -b feature/new-api
# ... edit files ...
git commit -m "New API for solver"

# 2. Adapt CTModels
cd ../CTModels.jl
git checkout -b feature/adapt-to-new-api
# ... update code to use new CTBase API ...
git commit -m "Adapt to new CTBase API"

# 3. Test everything together
cd ../OptimalControl.jl
julia --project -e 'using Pkg; Pkg.test()'

Everything works? Great! Both changes are compatible.

Tests fail? Fix immediately - you have all the code locally.

---

Handling Breaking Changes

Simple Process (Recommended for Small Teams)

Step 1: Announce
Post in team chat:

🚧 Working on breaking changes in CTBase today. 
Will update all packages. Don't merge to main until done!

Step 2: Make changes across all affected packages

cd CTBase.jl && git checkout -b epic/v0.17
# ... breaking changes ...

cd ../CTModels.jl && git checkout -b epic/v0.17  
# ... adapt code ...

cd ../CTParser.jl && git checkout -b epic/v0.17
# ... adapt code ...

# Continue for all affected packages

Step 3: Test everything together

cd OptimalControl.jl
julia --project=. -e 'using Pkg; Pkg.test()'

Step 4: When all works, push and create PRs

cd CTBase.jl && git push -u origin epic/v0.17
cd ../CTModels.jl && git push -u origin epic/v0.17
# ... etc

Step 5: Merge in dependency order

1. Merge CTBase PR first
2. Then CTModels (depends on CTBase)
3. Then CTParser, CTDirect, CTFlows
4. Finally OptimalControl

Step 6: Register new versions

Register in the same order:

1. @JuliaRegistrator register on CTBase → wait for registration
2. Register CTModels (after CTBase is available)
3. Register others
4. Register OptimalControl last

Timeline: Can complete in a few hours (not weeks!)

---

Version Management

During Development: Don't Touch Versions

While actively developing:

• ❌ Don't bump version numbers
• ❌ Don't update compat constraints
• ❌ Don't register packages
• ✅ Just work freely across all packages

When Ready to Release

Only register when:

• Feature is complete and tested
• You want external users to have access
• You need a stable reference point

Process:

1. Merge all feature branches to main
2. Bump version numbers in Project.toml:
   • Patch (0.16.0 → 0.16.1): Bug fixes
   • Minor (0.16.0 → 0.17.0): Breaking changes (for v0.x)
   • Major (1.0.0 → 2.0.0): Breaking changes (for v1.x+)
3. Update compat constraints if needed
4. Register using @JuliaRegistrator register
5. Wait for registration to complete before registering dependents

Frequency: Weekly or monthly releases are fine. Don't release daily!

---

Common Tasks

Check What's in Development Mode

using Pkg
Pkg.status()

Look for (dev) markers to see which packages are using local code.

Switch Back to Registry Version

using Pkg
Pkg.free("CTBase")  # Use registry version of CTBase

To free all packages:

for pkg in ["CTBase", "CTModels", "CTParser", "CTDirect", "CTFlows"]
    Pkg.free(pkg)
end

Reset Everything (When Things Break)

using Pkg

# Free all developed packages
Pkg.free("CTBase")
Pkg.free("CTModels")
# ... etc

# Or nuclear option: delete Manifest and rebuild
rm("Manifest.toml")
Pkg.instantiate()

Pull Latest Changes from Team

cd ~/dev/control-toolbox

# Update all repositories
for dir in */; do
  cd "$dir"
  git pull origin main
  cd ..
done

Since everything is in development mode, all packages automatically use the updated code.

---

Team Coordination

Communication

Use team chat for:

• Announcing breaking changes
• Coordinating merges
• Asking for quick reviews
• Sharing progress

Example messages:

👋 Working on CTBase issue #42, ETA 2 hours
🚧 Breaking change incoming in CTBase, will update all packages
✅ All packages updated, ready for review
🎉 Released CTBase v0.17.0

Branch Strategy

Simple approach:

• Create feature branches: feature/description
• Create PRs to main
• Require 1-2 approvals (or self-merge if trusted)
• Delete branch after merge

For coordinated changes:

• Use the same branch name across repos: epic/v0.17
• Makes it clear which branches belong together
• Merge all together when ready

---

Example: Complete Feature Development

Scenario: Add new solver algorithm requiring changes in CTBase and CTModels

Monday Morning (1 hour)

09:00 - cd CTBase.jl && git checkout -b feature/new-solver
09:15 - # Implement new solver in CTBase
09:30 - julia --project -e 'using Pkg; Pkg.test()'  # Test CTBase
09:45 - cd ../CTModels.jl && git checkout -b feature/use-new-solver
10:00 - # Add function to use new solver
10:15 - cd ../OptimalControl.jl
10:30 - julia --project -e 'using Pkg; Pkg.test()'  # Test everything
10:45 - # Everything works? Great!

Monday Afternoon (30 min)

14:00 - # Push branches and create PRs
14:15 - # Quick team review
14:30 - # Merge both PRs

Later (when ready to release)

- Register CTBase v0.16.1 (new algorithm)
- Register CTModels v0.10.3 (uses new algorithm)

Total active time: 1.5 hours. Feature is developed, tested, and merged.

---

Advantages of This Workflow

✅ Fast iteration - No waiting for registrations or CI
✅ Simple - Easy to understand and explain
✅ Flexible - Easy to experiment and refactor
✅ Julia-native - Uses Pkg.develop() as intended
✅ Safe - Test everything together before releasing
✅ Low overhead - Minimal coordination needed

---

When to Evolve This Workflow

This workflow is ideal for:

• Small teams (3-10 developers)
• 5-15 interdependent packages
• Packages under active development
• Internal or small user base

Consider more structure when:

• Team grows beyond 10 people
• You have many external users
• Packages become very popular
• Breaking others' code becomes a real risk

But until then: Keep it simple! 🚀

---

Quick Reference

Initial Setup

# Clone all repos
git clone https://github.com/control-toolbox/CTBase.jl.git
# ... etc

# Set up development
cd OptimalControl.jl
julia --project -e 'using Pkg; Pkg.develop(path="../CTBase.jl")'

Daily Development

# Work in any package, changes immediately available everywhere
cd CTBase.jl  # edit code
cd ../OptimalControl.jl  # test with changes
julia --project -e 'using Pkg; Pkg.test()'

Check Status

using Pkg
Pkg.status()  # Look for (dev) markers

Release

# 1. Merge feature branches
# 2. Bump versions in Project.toml
# 3. Register with @JuliaRegistrator register
# 4. Register in dependency order

---

Getting Help

• Julia Pkg docs: https://pkgdocs.julialang.org/
• Team chat: Ask questions anytime!
• This wiki: Check other guides

Happy developing! 🎉

[ocots]

Diamond Dependency Problem: Blocked vs Unblocked

Graph Structure

    A
   ╱ ╲
  B   D
   ╲ ╱
    C

Dependencies:

• A depends on B, C, and D
• B depends on C
• D depends on C
• C is the base package (no dependencies)

---

Julia Version Resolution Rules

Rule 1: Pre-release Filtering (Most Important!)

By default, pre-releases are completely IGNORED during resolution.

# Available versions: C v1, v2-beta.1
# Compat declaration: C = "1"

# Resolution process:
# 1. Filter: Remove all pre-releases → candidates = {v1}
# 2. Resolve: Only v1 is considered
# Result: v1

Pre-releases only become candidates if EXPLICITLY allowed:

# Available versions: C v1, v2-beta.1
# Compat declaration: C = "1, 2.0.0-"  ← Explicit mention of pre-releases

# Resolution process:
# 1. Filter: Keep pre-releases (explicitly allowed) → candidates = {v1, v2-beta.1}
# 2. Resolve: Both are now candidates
# 3. Select: Prefer stable over pre-release → v1
# Result: v1

Rule 2: Version Ordering

Once filtered, versions follow: v1 < v2-alpha < v2-beta < v2-rc < v2

Rule 3: Intersection

The resolver finds the intersection of all constraints:

Package A requires: C ∈ {v1, v2-}
Package B requires: C ∈ {v2-}
Intersection: {v1, v2-} ∩ {v2-} = {v2-}

Rule 4: Stable Preference

When intersection contains both stable and pre-release versions, prefer stable.

Intersection = {v1, v2-beta.1} → Choose v1 (stable preferred)
Intersection = {v2-beta.1} → Choose v2-beta.1 (only option)

Rule 5: Maximum Selection

Within the same stability class, choose the highest version.

Intersection = {v2-beta.1, v2-beta.2} → Choose v2-beta.2
Intersection = {v1.0, v1.1} → Choose v1.1

---

The Key Insight: Why Betas Matter

Betas don't get chosen over stables.

Betas create resolution PATHS that didn't exist before.

Without beta:

Intersection = ∅ → UNSATISFIABLE ❌

With beta:

Intersection = {v2-beta} → SATISFIABLE ✅

The beta is used ONLY when it's the ONLY way to satisfy all constraints.

---

Scenario: C Breaking Change (v1 → v2)

Initial State (All Stable)

Available versions:

C: v1
B: v1 (requires C=v1)
D: v1 (requires C=v1)
A: v1 (requires B=v1, D=v1, C=v1)

Compat declarations:

Compat(A, v1) = {B ↦ {v1}, D ↦ {v1}, C ↦ {v1}}
Compat(B, v1) = {C ↦ {v1}}
Compat(D, v1) = {C ↦ {v1}}
Compat(C, v1) = {}

Resolution for A:

A requires: B=v1, D=v1, C=v1
B requires: C=v1
D requires: C=v1

Intersection on C: {v1} ∩ {v1} ∩ {v1} = {v1} ✅

Configuration: {(A,v1), (B,v1), (D,v1), (C,v1)} ✅

---

Problem 1: BLOCKED - Without Beta Strategy

Step 1: C v2 Released (Breaking Change)

Available versions:

C: v1, v2
B: v1 (still requires C=v1)
D: v1 (still requires C=v1)
A: v1 (still requires B=v1, D=v1, C=v1)

Note: v2 is stable, so it's not filtered out. But constraints prevent its use.

Step 2: B Updated to Use C v2

Create B v2 that requires C v2:

B: v1 (requires C=v1), v2 (requires C=v2)

Compat:

Compat(B, v2) = {C ↦ {v2}}  # Breaking: now requires v2

Step 3: Try to Test A with New B v2

Attempt configuration: {(A,v1), (B,v2), (D,v1), (C,?)}

Constraints on C:

From A v1: C ∈ {v1}
From B v2: C ∈ {v2}
From D v1: C ∈ {v1}

Intersection: {v1} ∩ {v2} ∩ {v1} = ∅

Result: ❌ UNSATISFIABLE - Cannot test A with B v2!

Why? A and D require C v1, but B v2 requires C v2. No common version exists.

---

Solution: UNBLOCKED - With Beta Strategy

Phase 1: Preventive Widening

BEFORE releasing C v2, widen all compat constraints:

Compat(A, v1.1) = {B ↦ {v1}, D ↦ {v1}, C ↦ {v1, v2-}}  # Accept v1 AND v2 betas
Compat(B, v1.1) = {C ↦ {v1, v2-}}                       # Accept both
Compat(D, v1.1) = {C ↦ {v1, v2-}}                       # Accept both

Register new versions:

A: v1, v1.1 (widened compat)
B: v1, v1.1 (widened compat)
D: v1, v1.1 (widened compat)

User installing A:

using Pkg
Pkg.add("A")

# Resolution:
# 1. A v1.1 requires: B ∈ {v1}, D ∈ {v1}, C ∈ {v1, v2-}
# 2. B v1.1 requires: C ∈ {v1, v2-}
# 3. D v1.1 requires: C ∈ {v1, v2-}
# 4. Intersection: {v1, v2-} ∩ {v1, v2-} ∩ {v1, v2-} = {v1, v2-}
# 5. Filter pre-releases: v2- candidates don't exist yet (only widened compat)
# 6. Choose: v1 (stable, only concrete option)

Result: A v1.1, B v1.1, D v1.1, C v1 ✅

No disruption for users!

Phase 2: Release C v2-beta

C: v1, v2-beta.1

User installing A:

Pkg.add("A")

# Resolution:
# 1. Constraints create intersection: {v1, v2-}
# 2. Candidates after filtering: {v1, v2-beta.1}  ← beta now allowed!
# 3. Prefer stable: v1 chosen over v2-beta.1
# Result: A v1.1, B v1.1, D v1.1, C v1 ✅

Why? Even though v2-beta.1 is in the intersection, the resolver prefers stable v1.

Key point: User still gets v1, but now v2-beta.1 is "ready" as a fallback path.

Phase 3: Update B to Use C v2-beta

Release B v2:

B: v1, v1.1, v2 (requires C=v2-beta)

Compat(B, v2) = {C ↦ {v2-}}  # Requires beta explicitly

Phase 4: Test A with B v2 (NOW IT WORKS!)

Attempt configuration: {(A,v1.1), (B,v2), (D,v1.1), (C,?)}

Constraints on C:

From A v1.1: C ∈ {v1, v2-}   ← Widened!
From B v2:   C ∈ {v2-}       ← Forces beta
From D v1.1: C ∈ {v1, v2-}   ← Widened!

Intersection: {v1, v2-} ∩ {v2-} ∩ {v1, v2-} = {v2-}

Filtering and selection:

Intersection = {v2-}
Available candidates matching {v2-}: {v2-beta.1}
Stable preference: N/A (no stable in intersection)
Choose maximum pre-release: v2-beta.1

Result: ✅ SATISFIABLE

Configuration: {(A,v1.1), (B,v2), (D,v1.1), (C,v2-beta.1)} ✅

Now we can test A with the new B! 🎉

Why it works:

• Without widening: intersection = ∅ (no v2-beta allowed)
• With widening: intersection = {v2-beta} (path exists!)
• Beta is USED because it's the ONLY option in the intersection

Phase 5: Also Update D (Same Process)

D: v1, v1.1, v2 (requires C=v2-beta)

Test A with both B v2 and D v2:

Configuration: {(A,v1.1), (B,v2), (D,v2), (C,?)}

Constraints:

From A v1.1: C ∈ {v1, v2-}
From B v2:   C ∈ {v2-}
From D v2:   C ∈ {v2-}

Intersection: {v1, v2-} ∩ {v2-} ∩ {v2-} = {v2-}
Available: {v2-beta.1}

Result: ✅ {(A,v1.1), (B,v2), (D,v2), (C,v2-beta.1)}

Phase 6: Release C v2 Stable

C: v1, v2-beta.1, v2

User installing A with old dependencies:

Pkg.add("A")

# A v1.1 + B v1.1 + D v1.1
# Intersection: {v1, v2-}
# Available: {v1, v2-beta.1, v2}
# Prefer stable: v1 chosen ✅

User installing A with new B v2 and D v2:

# A v1.1 + B v2 + D v2
# Intersection: {v2-}
# Available matching {v2-}: {v2-beta.1, v2}
# v2 is stable pre-release → Choose v2 ✅

Everything works! ✅

---

Key Comparison

Without Beta Strategy (BLOCKED)

Timeline:
1. C v2 released (stable)
2. B v2 requires C v2
3. Try to test A with B v2:
   - A requires C ∈ {v1}
   - B requires C ∈ {v2}
   - Intersection = ∅
   - ❌ UNSATISFIABLE

Problem: No resolution path exists!

With Beta Strategy (UNBLOCKED)

Timeline:
1. Widen compat in A, B, D: C ∈ {v1, v2-} (preventive)
2. Release C v2-beta
3. Release B v2 (requires C ∈ {v2-})
4. Test A with B v2:
   - A requires C ∈ {v1, v2-}
   - B requires C ∈ {v2-}
   - Intersection = {v2-}
   - Available: {v2-beta.1}
   - ✅ SATISFIABLE with v2-beta.1

Problem solved: Beta creates a resolution path!

---

Visual Resolution Steps

Blocked Case (No Beta)

State: C has v1, v2 (both stable)
  
A v1 requires: C ∈ {v1}        ┐
B v2 requires: C ∈ {v2}        ├─ Intersection = ∅
D v1 requires: C ∈ {v1}        ┘

❌ NO SOLUTION

Unblocked Case (With Beta)

State: C has v1, v2-beta.1

After widening:
A v1.1 requires: C ∈ {v1, v2-}  ┐
B v2 requires:   C ∈ {v2-}      ├─ Intersection = {v2-}
D v1.1 requires: C ∈ {v1, v2-}  ┘

Available matching {v2-}: {v2-beta.1}
✅ SOLUTION: C = v2-beta.1

---

Why This Strategy Works: The Complete Picture

1. Widening Creates Potential Paths

Before: C ∈ {v1}           → Only v1 can satisfy
After:  C ∈ {v1, v2-}      → v1 OR v2- can satisfy

2. Betas Enable New Intersections

Without beta:
  Intersection of {v1} and {v2} = ∅

With beta:
  Intersection of {v1, v2-} and {v2-} = {v2-} ✓

3. Stable Preference Protects Users

User installs A alone:
  Intersection = {v1, v2-}
  Available = {v1, v2-beta.1}
  Preference = stable
  Result = v1 ✅ (protected!)

Developer tests A with B v2:
  Intersection = {v2-}  ← Forced by B v2
  Available = {v2-beta.1}
  No stable available
  Result = v2-beta.1 ✅ (testing works!)

4. Gradual Migration

Time 1: Everyone on v1 (stable)
Time 2: Beta available, but users still on v1
Time 3: Some packages migrate to v2-beta
Time 4: v2 stable released
Time 5: Everyone migrates to v2

---

Summary

The Paradox Resolved

You asked: "If resolver takes maximum, why don't users get beta?"

Answer: The resolver does NOT simply take maximum. It follows this order:

1. Filter by compat (ignore pre-releases unless explicitly allowed)
2. Find intersection (versions satisfying all constraints)
3. Prefer stable (choose stable over pre-release when both available)
4. Take maximum (within stability class)

Result:

• Intersection = {v1, v2-beta} → Choose v1 (stable preferred)
• Intersection = {v2-beta} → Choose v2-beta (only option)

Why Betas Are Essential

Without betas:

• Intersection can be empty → Unsatisfiable
• No way to test packages together
• Must coordinate all changes simultaneously

With betas:

• Create non-empty intersections → Satisfiable
• Enable progressive testing
• Allow parallel development
• Users protected by stable preference

Key insight: Betas are fallback paths used only when necessary, not preferred alternatives.

[ocots]

See new repo https://github.com/control-toolbox/DevelopmentGuide.