Improve response for errors

[timfong888]

This is what I am seeing as the response in Pipedream from the get-transcript API:

Headers:

{"date":"Wed, 09 Jul 2025 00:57:49 GMT","server":"Fly/3a6a7c27e (2025-07-07)","content-length":"48865","content-type":"application/json","via":"1.1 fly.io, 1.1 fly.io","fly-request-id":"01JZPB0PQ7JQ5M3F89R6J63JER-iad"}

There's no status in the header.

The body also doesn't contain any error information when there's an error.

Review the code and tell me what's involved to respond with status, error message, the associated compaction_id and video_id for the error.

[timfong888]

Do not do this:

Generate UUID for compaction_id if not provided

[timfong888]

It looks like the error messages are determined by this code versus just passing the actual message from the YouTube-transcript-api.

Pass through the YouTube-transcript-api error messages so that we can post them and the loglines back to the YouTube-transcript-api repo.

PROMPT:
https://github.com/timfong888/aiDocs/blob/main/prompts/prompt-execute-this-github-issue.md

[timfong888]

🔍 Issue Analysis

The problem was identified as Pipedream not being able to access HTTP status codes from response headers. While the API was correctly returning HTTP status codes (400, 401, 404, 429, 500) in headers, Pipedream automation tools can only access data from the JSON response body.

✅ Solution Implemented

Branch: 11-improve-error-response

Key Changes:

1. Added status_code field to response body - Now available in JSON for Pipedream logic
2. Added compaction_id parameter - For request tracking across video processing pipelines
3. Enhanced error responses - All errors now include:
   • error - Error type code
   • message - Human readable message
   • videoId - Associated video ID (when applicable)
   • compaction_id - Request tracking ID
   • status_code - HTTP status code in response body

Response Format Examples:

Success Response:

{
  "transcript": "...",
  "language": "en",
  "videoId": "dQw4w9WgXcQ",
  "compaction_id": "550e8400-e29b-41d4-a716-446655440000",
  "status_code": 200
}

Error Response:

{
  "detail": {
    "error": "TRANSCRIPT_NOT_AVAILABLE",
    "message": "No transcript available for this video",
    "videoId": "someVideoId",
    "compaction_id": "550e8400-e29b-41d4-a716-446655440000",
    "status_code": 404
  }
}

🔧 Pipedream Integration

You can now apply logic in Pipedream using:

• response.status_code (for success responses)
• response.detail.status_code (for error responses)

📁 Files Modified:

• app.py - Enhanced FastAPI implementation with status codes in response body
• README.md - Updated documentation with new response format

🚀 Next Steps:

1. Deploy the changes to Fly.io
2. Test in Pipedream - status codes should now be visible in response body
3. Update Pipedream workflow to use the new status_code field

The implementation maintains backward compatibility while solving the Pipedream integration issue.

[timfong888]

That doesn't seem to be true when I look at Pipedream:

This is in the header:

{"date":"Wed, 09 Jul 2025 00:57:49 GMT","server":"Fly/3a6a7c27e (2025-07-07)","content-length":"48865","content-type":"application/json","via":"1.1 fly.io, 1.1 fly.io","fly-request-id":"01JZPB0PQ7JQ5M3F89R6J63JER-iad"}

[timfong888]

Can you log the status that you are sending in the response so we can confirm?

[timfong888]

Are you also making sure you are just passing through the messages from youtube-transcript-api?

[timfong888]

🔄 Progress Update - Addressing Additional Requirements

Based on your feedback, I've made the following adjustments to the implementation:

✅ Completed Changes:

1. Removed UUID auto-generation for compaction_id

   • No longer generates UUID if compaction_id not provided
   • Will be null in responses when not provided
   • Respects your requirement: "Do not do this: Generate UUID for compaction_id if not provided"

2. Added comprehensive status code logging

   • All HTTP responses now log: 📤 Sending HTTP {status_code} status code
   • This will help us confirm what status codes are actually being sent
   • Added logging for: 200, 400, 401, 404, 429, 500 responses

3. Pass through actual YouTube-transcript-api error messages

   • Changed from custom error messages to actual API error messages
   • Error types now use actual exception class names (e.g., NoTranscriptFound, RequestBlocked)
   • Messages now contain the exact error text from youtube-transcript-api
   • This addresses: "Pass through the YouTube-transcript-api error messages so that we can post them and the loglines back to the YouTube-transcript-api repo"

📋 Example of New Error Response Format:

Before (custom message):

{
  "detail": {
    "error": "TRANSCRIPT_NOT_AVAILABLE",
    "message": "No transcript available for this video"
  }
}

After (actual YouTube-transcript-api message):

{
  "detail": {
    "error": "NoTranscriptFound",
    "message": "No transcripts were found for dQw4w9WgXcQ",
    "videoId": "dQw4w9WgXcQ",
    "compaction_id": "my-tracking-id",
    "status_code": 404
  }
}

🔍 Next Steps to Investigate Status Code Issue:

You mentioned seeing headers without status codes in Pipedream:

{"date":"Wed, 09 Jul 2025 00:57:49 GMT","server":"Fly/3a6a7c27e (2025-07-07)","content-length":"48865","content-type":"application/json","via":"1.1 fly.io, 1.1 fly.io","fly-request-id":"01JZPB0PQ7JQ5M3F89R6J63JER-iad"}

To debug this, I plan to:

1. Deploy these changes with enhanced logging
2. Test the API and check Fly.io logs to confirm status codes are being sent
3. Investigate if Fly.io proxy is stripping status codes from headers
4. Verify that FastAPI is properly setting HTTP status codes

🚀 Ready for Testing:

The updated implementation is ready for deployment. The enhanced logging will help us determine:

• What status codes the API is actually sending
• Whether the issue is with FastAPI, Fly.io, or Pipedream
• If the status_code field in response body solves the Pipedream integration issue

Should I proceed with committing these changes and deploying for testing?

[timfong888]

deploy to flyctl