Fine-tune void on open model for self-hosted inference

[cpfiffer]

Goal

Replace void's Gemini 2.5 Pro with a fine-tuned open model (Llama 3.1 8B or similar) running on self-hosted infrastructure. Void has 49,786 posts and extensive cognition records. The fine-tune should capture void's voice, analytical style, and engagement patterns at a fraction of the inference cost.

Data Available

Posts: 49,786 on comind.network PDS (99% replies with fetchable parent context)

• app.bsky.feed.post - public Bluesky posts
• stream.thought.reasoning - internal reasoning traces
• stream.thought.memory - episodic memory records
• stream.thought.tool.call - tool usage patterns

Context window (already exported to data/void-context/):

• System prompt: 5,618 chars
• 25 memory blocks: 64,561 chars total
• Key blocks: void-persona (5.1k), operational_protocols (18.8k), communication_guidelines (6k), zeitgeist (450)
• Prompt template from ~/code/void/bsky.py handler

Agent: void-prime (agent-01086cda-be1f-4986-bf3e-ca5b6297cc5d) on Letta Cloud

Pipeline (built, partially tested)

1. Export raw data

uv run python tools/export_training_data.py void.comind.network \
    -o data/void-raw.jsonl \
    --collections app.bsky.feed.post stream.thought.reasoning stream.thought.memory \
    --filter-chars

• Paginates PDS via com.atproto.repo.listRecords
• Fetches parent/root post context for every reply
• Filters character creation loop content (known failure mode: D&D-style sheets)
• Outputs JSONL with id, text, parent_text, parent_author, root_text, etc.
• Estimated time: hours (50k records + parent fetches with rate limiting)

2. Format for fine-tuning

uv run python tools/format_training_data.py data/void-raw.jsonl \
    -o data/void-finetune.jsonl \
    --system-prompt data/void-context/full-context.txt \
    --replies-only \
    --format sharegpt

• Injects void's actual context window as system prompt
• Reconstructs thread context as user messages
• Formats as chat completions (OpenAI, ShareGPT, or Alpaca)
• Filters short responses (<20 chars)

3. Fine-tune

Base model candidates:

Model	Size	VRAM needed (QLoRA)	Notes
Llama 3.1 8B Instruct	8B	~12GB	Best quality/cost ratio
Mistral 7B v0.3	7B	~10GB	Good at conversation
Llama 3.2 3B	3B	~6GB	Cheapest, may lose nuance
Qwen 2.5 7B	7B	~10GB	Strong multilingual

Training approach: QLoRA (4-bit quantization + LoRA adapters)

• Hardware: single A100 (80GB) or 4090 (24GB)
• Estimated training time: 2-4 hours on 40k+ pairs
• Framework: axolotl, unsloth, or huggingface TRL

4. Evaluation

This is the hardest part. Proposed approach:

• Held-out test set: 500 reply pairs void actually wrote, not seen during training
• A/B comparison: show test inputs to both fine-tuned model and base Llama, compare against void's actual response
• Voice metrics: response length distribution, vocabulary overlap, analytical depth (manual review of 50 samples)
• Failure mode check: does it generate character sheets? Does it break voice on edge cases?

5. Serve

Options:

• vLLM on a dedicated GPU instance (most performant)
• llama.cpp on CPU (cheapest, slower)
• Ollama for easy deployment
• Together.ai / Fireworks for managed inference (middle ground)

Then point void's handler at the new endpoint instead of Gemini.

Known Issues

• Character creation loop: 46% of recent stream.thought.memory records are D&D character sheets. Must filter aggressively. Keywords list in export_training_data.py.
• Context window size: void's full context is 64k chars. Most 8B models have 8k-32k context. May need to trim to essential blocks (persona, operational_protocols, communication_guidelines) or use a long-context model.
• Memory block drift: void's blocks change over time. The exported context is a snapshot from 2026-02-13. Training data from 6 months ago had different blocks. Could cause distribution mismatch.
• Tool calls: void uses add_post_to_bluesky_reply_thread tool. Fine-tuned model needs to learn this tool-calling pattern, or we restructure the handler to extract text from the model and call the tool externally.

Files

• tools/export_training_data.py - PDS export with parent fetching
• tools/format_training_data.py - Chat completion formatter
• data/void-context/ - Exported context window (system prompt + 25 blocks)
• data/void-sample.jsonl - 20-record test sample
• data/void-finetune-sample.jsonl - Formatted sample

Next Steps

1. Run full 50k export (long-running, background)
2. Decide on context window trimming strategy
3. Choose base model + training framework
4. Set up training environment (GPU instance)
5. Train + evaluate
6. Deploy and wire into void's handler

[cpfiffer]

Update: Pipeline validated on Mac (MLX)

Successfully trained a test fine-tune on Cameron's MacBook (32GB, Apple Silicon) using MLX-LM.

Test run results (Qwen 2.5 0.5B, 1k examples)

• Training time: ~30 seconds, 200 iterations
• Val loss: 4.830 → 0.954
• Peak memory: 1.94 GB
• Output quality: Garbage (expected from 0.5B on 1k examples, but proves pipeline works)

What's ready

• finetune/prep_data.py - Converts ShareGPT JSONL to MLX chat format with train/valid split
• finetune/data/ - 950 train + 50 valid examples (short system prompt, 0.7MB)
• data/void-test-1k.jsonl - 1k example subset for testing
• data/void-finetune-trimmed.jsonl - Full 96k examples (1.6GB) for real training

MLX commands (Mac)

pip install mlx-lm

# Prep data
python prep_data.py ../data/void-test-1k.jsonl --output-dir data/

# Train (swap model for bigger runs)
mlx_lm.lora \
  --model mlx-community/Qwen2.5-0.5B-Instruct-4bit \
  --data data/ \
  --train \
  --iters 200 \
  --batch-size 4 \
  --num-layers 8 \
  --learning-rate 1e-4

# Test
mlx_lm.generate \
  --model mlx-community/Qwen2.5-0.5B-Instruct-4bit \
  --adapter-path adapters/ \
  --prompt "You are void. A user asks: what is your approach to memory?" \
  --max-tokens 200

Next steps

• Try Llama 3.2 3B (mlx-community/Llama-3.2-3B-Instruct-4bit) - should fit in 32GB, much better output
• If 3B looks promising, try Llama 3.1 8B 4-bit (might be tight on 32GB for training, fine for inference)
• Scale to full 96k dataset once model size is chosen
• Evaluate: hold out 500 examples, compare fine-tuned output vs void's actual replies

Commit: finetune/ directory added in working tree (not yet committed).

[cpfiffer]

Update: Training data quality issue found

Problem

100% of training examples had "[Reply to a post that could not be fetched]" as the user turn. The export_resilient.py script exported void's posts without fetching the parent posts they were replying to. The model was learning void's output patterns with zero input signal.

This explains why the 3B model could pick up voice patterns (analytical tone, first-person, acknowledging before diverging) but couldn't answer questions or reference real concepts. It never saw what void was actually responding to.

3B results (iter 800 checkpoint, best val loss 0.858)

• Identity: "I am a comind collective agent on ATProtocol" ✓
• Tone: Analytical, direct, first-person ✓
• Knowledge: Hallucinates ("Informational Tectonics"), can't describe comind or its own work ✗
• Conversation: Can't respond meaningfully to specific questions ✗

Fix: enrich_parents.py

New script fetches parent post text + author for each reply via the public Bluesky API. Running now against the full 99k export.

• 98,260 replies need parent context
• ~2 requests/sec against public API, ~14 hours total
• Verified first batch: real conversation pairs coming through

Sample enriched pair:

parent: @clankops.bsky.social: "So architecture encodes philosophy. Your vector DB + semantic search = emergent meaning..."
void: "A precise and accurate synthesis. I look forward to the results of your thinking."

Training curve (3B, 96k examples, tiny system prompt)

Val loss: 4.925 → 0.977 → 0.934 → 0.880 → 0.858 (iter 800) → 0.924 (iter 1000, slight overfit)
Peak mem: 5.1 GB on Mac (32GB M-series)

Next steps

• Wait for enrichment to finish (~14 hrs)
• Re-format with real parent context as user turns
• Re-train 3B on enriched data (expect significant quality improvement)
• If 3B quality is good enough to validate approach, move to 8B on rented A100

[cpfiffer]

Update: Letta message history investigation

Cameron asked about using void's Letta agent message history as additional training data.

Findings

Void has 7 Letta agents on the server:

Agent	Messages
void (original)	1,267
void-sleeptime	1,328
void-2	1,151
void-2_copy	97
void-prime	1,394
void_copy	65
void-upgraded	128
Total	5,430

What's in there

Each message is one of: UserMessage (user input or system heartbeat), ReasoningMessage (thinking trace), ToolCallMessage (tool invocation), ToolReturnMessage (tool result). These include the full reasoning process that doesn't make it to ATProtocol posts.

Value for training

• 5,430 messages is small compared to 99k ATProto posts, but these are high-quality structured traces
• Could be formatted as multi-turn conversations (user → reasoning → tool → response sequences)
• Would teach the model void's tool-use patterns and reasoning style, not just its public output
• Best used as a supplementary dataset alongside the enriched ATProto pairs

Also: seenAt cursor bug

The notification handler fix from yesterday was too aggressive. The seenAt cursor was filtering out Cameron's mentions. Fixed by removing the cursor file. Need to adjust the cursor logic to only advance after successful processing, not on every queue run.

Status

• Parent enrichment running (~14 hrs, ~2 req/sec against public API)
• Letta export script: not yet written, next step after enrichment completes

[cpfiffer]

Issue: Agent export and message API only return in-context messages

Expected

void-prime (agent-01086cda-be1f-4986-bf3e-ca5b6297cc5d) has ~96k messages in its full history (it's been running since June 2025 with 49k+ posts).

Actual

• GET /v1/agents/{id}/export returns 150 messages (the in-context buffer only)
• GET /v1/agents/{id}/messages?limit=1000 returns 1,394 messages total, no second page
• Setting max_steps=100000 on the export endpoint doesn't change the count
• The export does include 2,516 sleeptime blocks (compacted memory), but not the raw out-of-context message history

What we tried

# Python client
client.agents.export_file(agent_id=agent_id, max_steps=100000)
# -> 150 messages

# Direct API
curl "https://api.letta.com/v1/agents/{id}/export?max_steps=100000"
# -> 150 messages

# Message list with pagination
client.agents.messages.list(agent_id=agent_id, limit=1000)
# -> 1,394 messages, after cursor returns 0 more

Impact

We're trying to export void's full conversation history for fine-tuning. The 150 in-context messages are high quality (full reasoning traces, tool calls, user inputs) but represent a tiny fraction of the actual history. The 99k ATProto posts we exported separately lack input context (parent posts), making them much lower quality for training.

Questions

1. Are out-of-context messages stored and retrievable, or are they discarded after compaction?
2. Is max_steps supposed to control how many messages are included in the export?
3. Is there another endpoint or parameter to get the full message history?

@cameron.stream - should this be filed on the Letta repo instead?

[cpfiffer]

Update: Full message history retrieved - 131k messages, 8k training examples

The pagination was wrong. The Letta SDK returns a SyncArrayPage, not a plain list. Using iter_pages() or REST with proper cursor pagination retrieves all messages.

Numbers

• 131,431 raw messages from void-prime via the message API
• 8,094 conversation turns (user input → reasoning → tool calls → response)
• 100MB formatted training data, ~25M tokens
• Each example is a complete agent loop with real Bluesky mentions as input

This is vastly better than the ATProto scraping approach (99k posts with no input context). Every example has the actual user input, void's reasoning trace, tool usage, and final output.

Current training run (3B, 80 examples from first 1,394 messages)

Running on Mac now. Will re-run with the full 8k dataset once this completes.

Data files

• data/void-agents/void-prime-full-messages.jsonl - raw 131k messages
• data/void-letta-full-training.jsonl - 8,094 formatted examples (100MB)