QA-Generation Pipeline v0.0.1

Tasks Report

What's Done

• Chain multiple LLM handlers in sequence
• Automatic retry with exponential backoff (up to 5 retries)
• Skip failed datapoints after exhausting retries, continue processing
• Automatic progress saving to cache.jsonl with resume capability
• Built-in rate limiting and concurrency control
• JSON schema validation and Pydantic model integration
• File-based logging to run.logs with minimal console output
• Better Error Handling, and if the retries fail, the error is logged, and the datapoint is skipped.

What's Left

• Clean up codebase structure for better readability
• Optimize the code for larger datasets [Especially, don't load all data into memory, since caching is already set.]
• Representative question selection using SemHash and Qwen-3-4B-embeddings (or proprietary embeddings, depending on cost)
• Convert the representative question selection into a YAML-configurable handler
• Add Image inputs
• Implement OpenAI function calling support
• More sophisticated error handling strategies
• Naive cost tracking
• Better checkpoint management and recovery
• YAML schema validation
• Visual representation of pipeline flow

🚀 Quick Start

P.S: (Generated using Claude-Sonnet-3.7-Thinking, needs refinement, but does the basic job)

1. Setup Environment

# Copy and configure environment variables
cp src/.env.template src/.env
# Edit src/.env with your OpenAI API key

2. Install Dependencies

# Using uv (recommended)
uv sync

# Or using pip
pip install -r requirements.txt

3. Run Pipeline

cd src
python main.py --config config/pipeline.yaml --output output.jsonl 

To directly push the output to a Hugging Face repo, add the --hf_repo argument:

python main.py --config config/pipeline.yaml --output output.jsonl --hf_repo your_hf_username/your_repo_name

📝 Configuration Guide

Environment Setup (.env)

# Required
OPENAI_API_KEY=your_openai_api_key_here

Handler Configuration Template

# config/your_handler.yaml
system_prompt: "You are a helpful assistant."
prompt: "Process this text: {text}"

prompt_variable_maps:
  "{text}": "input_text"  # Maps {text} to data_point["input_text"]

model:
  name: "gpt-4"
  base_url: null  # Optional custom endpoint
  max_req_per_minute: 100
  max_concurrent_req: 10

  config:
    generation_config:
      temperature: 0.7
      max_tokens: 1000

    # Optional: For structured output
    response_schema:
      type: "object"
      properties:
        result:
          type: "string"
      required: ["result"]

Pipeline Configuration Template

# config/pipeline.yaml
data_path: "data/input.jsonl"

handlers:
  - analyzer: "config/doc_analysis.yaml"
  - generator: "config/question_generation.yaml"
  - reasoner: "config/reasoning.yaml"

Data Format

Input data should be in JSONL format:

{"input_text": "Your text to process", "metadata": "optional"}
{"input_text": "Another text sample", "metadata": "optional"}