bug(channels): Telegram image media_type is application/octet-stream — breaks vision

[f-liva]

Bug Description

Photos sent via Telegram are downloaded and base64-encoded correctly, but the media_type is set to application/octet-stream instead of the actual image type (e.g. image/jpeg). This causes LLM providers to reject the image content, as they require a valid image MIME type.

Regression

This was supposed to be fixed as part of issue #362 and the related PR #365. The Telegram parsing and bridge download pipeline was added, but the media type detection has a bug that silently breaks vision.

Root Cause

In crates/openfang-channels/src/bridge.rs, download_image_to_blocks() (line ~818):

let content_type = resp
    .headers()
    .get("content-type")
    .and_then(|v| v.to_str().ok())
    .map(|ct| ct.split(';').next().unwrap_or(ct).trim().to_string());

let media_type = content_type.unwrap_or_else(|| {
    // fallback to URL extension detection
    ...
});

The code trusts the Content-Type header from Telegram's file download API. However, Telegram returns application/octet-stream for all file downloads via getFile. Since the header IS present (just wrong), the URL-extension fallback never runs.

The result: ContentBlock::Image { media_type: "application/octet-stream", data: "..." } is sent to the LLM, which rejects it because application/octet-stream is not a valid image MIME type.

Evidence

Session logs show: [image: application/octet-stream] — confirming the image data arrives but with wrong MIME type.

Proposed Fix

1. If Content-Type is application/octet-stream or another non-image type, fall through to secondary detection
2. Add magic-byte detection (JPEG FF D8, PNG 89 50 4E 47, etc.) as the most reliable method
3. Keep URL extension as tertiary fallback

Environment

• OpenFang v0.3.46
• Channel: Telegram (polling mode)
• LLM: qwen-plus (OpenAI-compatible API)

Related

• Telegram bridge: incoming photos/media are silently dropped #362 — Original issue (Telegram photos silently dropped)
• PR feat(telegram): support incoming photos, documents, voice and location #365 — My original PR implementing incoming media support (closed without merge)

[f-liva]

Real-world conversation excerpt — images not visible to agent

Adding a conversation excerpt from WhatsApp where the user sends expense receipt images, but the agent cannot see them due to the image handling pipeline issue:

---

User: Record these expenses
(image attached)

Agent: I do not see any image or expense list attached. Could you resend the photo or list the expenses for me?

User: How about now? Can you see it?
(image re-sent)

Agent: No, unfortunately I still cannot see any image.

---

Impact: The user is unable to use the most natural workflow (snap a photo of a receipt → agent records the expense). This forces a fallback to manually typing out each expense, which defeats the purpose of having a digital assistant.

Note: This specific excerpt is from WhatsApp channel, where the gateway strips image attachments entirely before forwarding. The same class of problem (images not reaching the agent vision model) affects Telegram via the application/octet-stream MIME type bug described in this issue.

[jaberjaber23]

Fixed in v0.3.47. Added three-tier image MIME detection in download_image_to_blocks(): trusted image/* Content-Type header, then magic byte sniffing (JPEG FF D8 FF, PNG 89 50 4E 47, GIF 47 49 46 38, WebP RIFF+WEBP), then URL extension fallback. Non-image Content-Types like application/octet-stream now fall through to secondary detection instead of being trusted.

[f-liva]

Unfortunately, the fix from PR #529 does not fully resolve the issue. After merging the media type detection changes into main and rebuilding, the agent still cannot see images sent via Telegram — the message arrives as text only, with no image attachment.

This suggests the problem runs deeper than media type detection and needs further investigation. We should also write automated tests to verify image handling end-to-end (receive image message → detect media → download → process) so we can catch regressions reliably.

I'll continue debugging and update this issue with findings.

[f-liva]

🔴 Bug still present — deeper root cause identified

The initial fix (PR #529, media_type correction) was necessary but not sufficient. Telegram images are still not reaching the agent in production. After a thorough investigation of the full pipeline, two additional bugs were found:

Root Cause 1: Vision model swap doesn't change the provider

In kernel.rs (~line 2268-2287), when an image arrives, the kernel correctly swaps the model to the configured vision_model (e.g. qwen-vl-plus), but does not update the provider field. It stays as claude-code, so the wrong driver is used:

// Before fix:
manifest.model.model = vision_model.clone();
// provider stays as "claude-code" → ClaudeCodeDriver is used instead of OpenAI/DashScope

Fix: Also swap the provider to match the vision model's provider:

manifest.model.model = vision_model.clone();
manifest.model.provider = self.config.default_model.provider.clone();

Root Cause 2: ClaudeCodeDriver silently drops all image content

In claude_code.rs, build_prompt() calls msg.content.text_content() which extracts only text from messages. All ContentBlock::Image and ContentBlock::ImageUrl blocks are silently discarded. Even if the provider issue were fixed and Claude Code were used intentionally, images would still be lost.

Fix: Updated ClaudeCodeDriver to:

• Extract images from content blocks (base64, data URI, and HTTP URL)
• Save them as temp files
• Pass them to the CLI via --files <path>

The actual flow (before fix)

Telegram photo → ContentBlock::ImageUrl ✅
  → Kernel swaps model to qwen-vl-plus ✅
  → Provider stays "claude-code" ❌
  → ClaudeCodeDriver.build_prompt() → text_content() only ❌
  → Agent receives empty/text-only message ❌

Status

Fix prepared across kernel.rs and claude_code.rs. Pending build and production test.

This issue should be reopened as the original fix did not resolve the problem.

[f-liva]

Resolved in #561

The original issue (#528) identified the media_type bug, but the full Telegram→Claude Code image pipeline had three additional bugs discovered during end-to-end testing:

1. Vision model selection: Kernel unconditionally swapped to a fallback vision_model even when the agent's model (e.g. Claude Opus) already supports vision — causing auth errors with misconfigured fallback models. Fixed with priority-based selection that checks the model catalog first.

2. Wrong CLI flag: Claude Code driver used --files (plural) — correct flag is --file (singular).

3. --file requires Files API: Even with the correct flag, --file expects Anthropic Files API IDs and CLAUDE_CODE_SESSION_ACCESS_TOKEN. Replaced entirely with @path syntax, which tells Claude Code CLI to read local files inline — no API tokens needed.

Result: Photos sent via Telegram now reach Claude Code agents and are correctly processed with vision capabilities. Verified working with Claude Opus agent.