Skip to content

Add Batch Scrape Support#31

Open
0xjmp wants to merge 2 commits intofirecrawl:mainfrom
0xjmp:main
Open

Add Batch Scrape Support#31
0xjmp wants to merge 2 commits intofirecrawl:mainfrom
0xjmp:main

Conversation

@0xjmp
Copy link
Copy Markdown

@0xjmp 0xjmp commented Nov 7, 2025

Implements the /v2/batch/scrape endpoint.

Changes:

  • Added types: BatchScrapeParams, BatchScrapeResponse, WebhookSpec, FormatSpec, ParserSpec, ActionSpec, LocationSpec, Viewport
  • Implemented BatchScrape() to start batch scrape jobs
  • Added tests covering basic usage, options, webhooks, and error cases
  • Improved error handling for empty responses
  • Added retry logic for 503 errors
  • The implementation follows the same patterns as other API methods and matches the OpenAPI specification.
$ go test -v -run TestBatchScrape                                                                                                              
=== RUN   TestBatchScrapeInvalidAPIKey
--- PASS: TestBatchScrapeInvalidAPIKey (0.34s)
=== RUN   TestBatchScrapeE2E
--- PASS: TestBatchScrapeE2E (0.16s)
=== RUN   TestBatchScrapeWithOptionsE2E
--- PASS: TestBatchScrapeWithOptionsE2E (0.10s)
=== RUN   TestBatchScrapeWithWebhookE2E
--- PASS: TestBatchScrapeWithWebhookE2E (0.35s)
=== RUN   TestBatchScrapeWithInvalidURLs
--- PASS: TestBatchScrapeWithInvalidURLs (0.09s)
=== RUN   TestBatchScrapeEmptyURLs
--- PASS: TestBatchScrapeEmptyURLs (0.00s)
=== RUN   TestBatchScrapeNilParams
--- PASS: TestBatchScrapeNilParams (0.00s)
PASS
ok      github.com/mendableai/firecrawl-go/v2   1.448s

@0xjmp
Copy link
Copy Markdown
Author

0xjmp commented Nov 7, 2025

@mogery I did notice this endpoint occasionally throw a 500 error

@0xjmp 0xjmp mentioned this pull request Nov 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant