Add option to balance jobs by example instead of by file

CLAUDE.md

@@ -0,0 +1,215 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+Flatware is a test parallelization tool for RSpec and Cucumber that significantly reduces test execution time by distributing tests across multiple worker processes using DRb (distributed Ruby) for inter-process communication.
+
+## Architecture Deep Dive
+
+### Core Components and Their Interactions
+
+#### 1. Worker Process Management (`lib/flatware/worker.rb`)
+- **Spawning**: Workers are created via `fork()` with unique IDs and `TEST_ENV_NUMBER` env vars
+- **Communication**: Uses DRb over Unix sockets (`drbunix:flatware-sink`)
+- **Retry Logic**: 10 retries with exponential backoff for DRb connections
+- **Lifecycle**: Pull-based model - workers request jobs from sink until receiving sentinel job
+- **Process Naming**: Sets `$0 = "flatware worker #{i}"` for easy identification
+
+#### 2. Sink Server (`lib/flatware/sink.rb`)
+Central coordinator implementing pull-based work distribution:
+- **Job Queue**: Maintains queue of jobs, distributes on worker request
+- **Worker Tracking**: Uses Set to track active workers
+- **Completion Detection**: Auto-stops when no workers remain and queue empty
+- **Signal Handling**: Graceful shutdown on INT/CLD signals
+- **Job Grouping**: Groups jobs by worker count using modulo for load balancing
+
+#### 3. Message Protocol
+DRb RPC methods between workers and sink:
+- `ready(worker_id)` - Worker requests work
+- `started(job)` - Worker reports job start
+- `finished(job)` - Worker reports job completion
+- `checkpoint(data)` - Send test results/progress
+- `progress(message)` - Real-time test output
+
+### Job Distribution Strategies
+
+#### FileJobBuilder (`lib/flatware/rspec/file_job_builder.rb`)
+Default strategy - distributes spec files:
+- Uses RSpec's persisted example status for time-based balancing
+- Greedy algorithm minimizes total runtime variance across workers
+- Untimed files distributed round-robin
+
+#### ExampleJobBuilder (`lib/flatware/rspec/example_job_builder.rb`)
+Fine-grained strategy - distributes individual examples:
+- Loads RSpec config in forked process to avoid pollution
+- Prioritizes longer-running examples first
+- Uses `within_forked_process` pattern for safe configuration loading
+
+### RSpec Integration Details
+
+#### Marshalable Object System (`lib/flatware/rspec/marshalable/`)
+Solves the problem of RSpec objects containing unmarshalable references:
+- `Example` - Strips RSpec examples to essential attributes
+- `ExecutionResult` - Serializes test results with custom exception handling
+- `SummaryNotification` - Aggregatable summary data across workers
+- Uses dynamic method generation for event handling
+
+#### Exception Serialization (`lib/flatware/serialized_exception.rb`)
+Handles unmarshalable exceptions by extracting:
+- Class name as string (not class object)
+- Message and backtrace
+- Recursively serializes cause chain
+
+#### Checkpoint System (`lib/flatware/rspec/checkpoint.rb`)
+- Aggregates test results from multiple workers
+- Implements `+` operator for combining checkpoints
+- Dynamically generates event handlers based on RSpec events
+
+### Process and Signal Management
+
+#### PID Tracking (`lib/flatware/pid.rb`)
+- Cross-platform PS parsing (Darwin vs Linux)
+- Process group tracking via pgid
+- Provides cleanup utilities for all flatware processes
+
+#### Signal Handling (`lib/flatware/sink/signal.rb`)
+- **INT**: Triggers graceful shutdown
+- **CLD**: Detects worker failures
+- Workers finish current jobs before exiting
+
+### Error Handling Patterns
+
+#### Worker Resilience
+- DRb connection retries with exponential backoff
+- Failed jobs remain in queue for other workers
+- Worker crashes detected via CLD signal
+
+#### Serialization Edge Cases
+- Exception cause chains properly handled
+- Class names stored as strings to avoid loading issues
+- Metadata filtered to exclude unmarshalable objects
+
+## Common Development Commands
+
+```bash
+# Run tests
+bundle exec rake spec     # Run RSpec tests
+bundle exec rake cucumber  # Run Cucumber tests
+bundle exec rake lint      # Run RuboCop linter
+bundle exec rake           # Run all checks (lint, spec, cucumber)
+
+# Test a single spec file
+bundle exec rspec spec/flatware/worker_spec.rb
+
+# Run Cucumber with specific tags
+bundle exec cucumber --tags "not @wip"
+
+# Build gems
+bundle exec rake build                    # Build all gems
+bundle exec rake build:flatware          # Build main gem
+bundle exec rake build:flatware-rspec    # Build RSpec runner
+bundle exec rake build:flatware-cucumber # Build Cucumber runner
+
+# Test flatware itself in development
+bundle exec flatware rspec
+bundle exec flatware cucumber
+
+# Debug with specific workers
+bundle exec flatware -w 2 rspec          # Use 2 workers
+bundle exec flatware --log rspec         # Enable debug logging
+
+# Use example-based distribution
+bundle exec flatware rspec --job-builder ExampleJobBuilder
+```
+
+## Database Configuration for Rails Apps
+When using Flatware with Rails, configure test databases with `TEST_ENV_NUMBER`:
+```yaml
+test:
+  database: foo_test<%=ENV['TEST_ENV_NUMBER']%>
+```
+
+Then prepare databases:
+```bash
+flatware fan rake db:test:prepare
+```
+
+## Flatware Configuration Hooks
+Create `spec/flatware_helper.rb` for lifecycle callbacks:
+```ruby
+Flatware.configure do |conf|
+  conf.before_fork do
+    require 'rails_helper'
+    ActiveRecord::Base.connection.disconnect!
+  end
+
+  conf.after_fork do |test_env_number|
+    config = ActiveRecord::Base.connection_db_config.configuration_hash
+    ActiveRecord::Base.establish_connection(
+      config.merge(database: config.fetch(:database) + test_env_number.to_s)
+    )
+  end
+end
+```
+
+## Key Implementation Patterns
+
+1. **Defensive Marshaling**: All cross-process objects explicitly made marshalable
+2. **Pull-based Distribution**: Workers request work (not pushed)
+3. **Sentinel Jobs**: Special jobs signal workers to shutdown cleanly
+4. **Broadcaster Pattern**: `method_missing` forwards to multiple formatters
+5. **Process Title Setting**: Each component sets `$0` for identification
+6. **Forked Configuration**: ExampleJobBuilder loads config in fork to avoid pollution
+7. **Dynamic Method Generation**: Checkpoint generates handlers from RSpec events
+
+## Debugging Tips
+
+### Process Inspection
+```bash
+ps aux | grep flatware  # See all flatware processes
+```
+
+### Common Issues
+1. **Segfault in PG gem**: Add `ENV["PGGSSENCMODE"] = "disable"` to flatware helper
+2. **ActiveRecord connection errors**: Ensure disconnect in before_fork, reconnect in after_fork
+3. **SimpleCov integration**: Call `SimpleCov.at_fork.call(test_env_number)` in after_fork
+
+### Testing Flatware Development
+Use Aruba for integration testing:
+1. Add `@no-clobber` tag to `features/flatware.feature`
+2. Run `cucumber features/flatware.feature`
+3. CD to `./tmp/aruba` where flatware is in PATH
+
+## Important File Locations
+
+### Core Logic
+- `lib/flatware/worker.rb` - Worker process implementation
+- `lib/flatware/sink.rb` - Central coordinator server
+- `lib/flatware/cli.rb` - Command-line interface (Thor)
+
+### RSpec Integration
+- `lib/flatware/rspec/` - RSpec-specific code
+- `lib/flatware/rspec/marshalable/` - Serialization layer
+- `lib/flatware/rspec/formatter.rb` - RSpec formatter integration
+
+### Job Distribution
+- `lib/flatware/rspec/file_job_builder.rb` - File-based distribution (default)
+- `lib/flatware/rspec/example_job_builder.rb` - Example-based distribution
+
+### IPC/Messaging
+- `lib/flatware/sink/client.rb` - Worker-to-sink communication
+- `lib/flatware/broadcaster.rb` - Event broadcasting to multiple formatters
+
+### Process Management
+- `lib/flatware/pid.rb` - Process tracking utilities
+- `lib/flatware/sink/signal.rb` - Signal handling
+
+## Non-Obvious Implementation Details
+
+- **Round-robin with Offset**: Untimed work uses calculated offsets for even distribution
+- **Retrying Helper**: Generic retry logic with exponential backoff used throughout
+- **Job Sentinels**: Empty jobs with `sentinel?` true trigger worker shutdown
+- **Configuration Isolation**: Each worker gets isolated RSpec configuration
+- **Checkpoint Aggregation**: Results combined via `+` operator across workers
+- **Process Group Management**: Uses `setpgrp` to manage related processes as group

features/rspec.feature

@@ -107,3 +107,46 @@ Feature: rspec task
       """
       0 examples, 0 failures, 1 error occurred outside of examples
       """
+
+  @non-zero
+  Scenario: example job builder
+    Given spec "a" contains:
+      """
+      describe "fail" do
+      it { expect(true).to eq false }
+      end
+      """
+    And spec "b" contains:
+      """
+      describe "pass" do
+      it { expect(true).to eq true }
+      end
+      """
+    When I run flatware with "rspec -l --job-builder=ExampleJobBuilder"
+    Then the output contains the following:
+      """
+      Run options: include {:ids=>{"./spec/a_spec.rb"=>["1:1"]}}
+      """
+    And the output contains the following:
+      """
+      Run options: include {:ids=>{"./spec/b_spec.rb"=>["1:1"]}}
+      """
+    And the output contains the following:
+      """
+      2 examples, 1 failure
+      """
+
+  @non-zero
+  Scenario: failure outside of examples with example job builder
+    Given the following spec:
+      """
+      throw :a_fit
+      describe 'fits' do
+        it('already threw one')
+      end
+      """
+    When I run flatware with "rspec --job-builder=ExampleJobBuilder"
+    Then the output contains the following line:
+      """
+      uncaught throw :a_fit
+      """

lib/flatware/rspec.rb

@@ -7,12 +7,14 @@
 module Flatware
   module RSpec
     require 'flatware/rspec/formatter'
-    require 'flatware/rspec/job_builder'
+    require 'flatware/rspec/file_job_builder'
+    require 'flatware/rspec/example_job_builder'
 
     module_function
 
-    def extract_jobs_from_args(args, workers:)
-      JobBuilder.new(args, workers: workers).jobs
+    def extract_jobs_from_args(args, workers:, job_builder:)
+      builder = const_get(job_builder)
+      builder.new(args, workers: workers).jobs
     end
 
     def runner

lib/flatware/rspec/cli.rb

@@ -13,18 +13,27 @@ class CLI
       type: :string,
       default: 'drbunix:flatware-sink'
     )
+    method_option(
+      'job-builder',
+      type: :string,
+      default: 'FileJobBuilder'
+    )
     desc 'rspec [FLATWARE_OPTS]', 'parallelizes rspec'
     def rspec(*rspec_args)
-      jobs = RSpec.extract_jobs_from_args rspec_args, workers: workers
-
-      formatter = Flatware::RSpec::Formatters::Console.new(
-        ::RSpec.configuration.output_stream,
-        deprecation_stream: ::RSpec.configuration.deprecation_stream
-      )
+      jobs = RSpec.extract_jobs_from_args rspec_args, workers: workers, job_builder: options['job-builder']
 
       Flatware.verbose = options[:log]
       Worker.spawn count: workers, runner: RSpec, sink: options['sink-endpoint']
       start_sink(jobs: jobs, workers: workers, formatter: formatter)
     end
+
+    private
+
+    def formatter
+      Flatware::RSpec::Formatters::Console.new(
+        ::RSpec.configuration.output_stream,
+        deprecation_stream: ::RSpec.configuration.deprecation_stream
+      )
+    end
   end
 end