Skip to content

Fix scraper timeout in Docker #47

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
atharvashahane28 merged 15 commits into from
Oct 9, 2025
Merged

Fix scraper timeout in Docker #47

atharvashahane28 merged 15 commits into from
Oct 9, 2025

Conversation

@atharvashahane28
Copy link
Collaborator

@atharvashahane28 atharvashahane28 commented Oct 9, 2025

This PR will:

  • Add a Dockerfile with instructions to build a Docker image that can run the data pipeline
  • Add necessary additional options to Selenium and Chromedriver
  • Add Shell & Python scripts to run the parts of the pipeline

Tests done:

  • Pipeline successfully runs in Windows, Linux, and MacOS
  • Pipeline successfully runs in Docker (locally)

atharvashahane28 and others added 15 commits September 9, 2025 16:29
@atharvashahane28
fix: decrease sleep time for poultry scraper
0f58e10
@atharvashahane28
fix: decrease sleep time for bovine scraper
92aeb91
@atharvashahane28
fix: increase WebDriverWait timeout period in poultry scraper
7ce53ee
@atharvashahane28
fix: increase WebDriverWait timeout period in bovine scraper
2d0bbc0
@atharvashahane28
Merge branch 'main' into fix/42-scraper-timeout
9f5a878
@atharvashahane28
style: wording in shell pipeline
2e6acf9
@atharvashahane28
feat: headless option in selenium
b89f300
@swetha4444
feat: add Dockerfile and container scripts for headless Chrome operation
163de87
@swetha4444
feat: update Chrome options for scrapers to improve headless operation
4c6a510
@swetha4444
refactor: remove outdated source commands from flow.R
8b692a1
@atharvashahane28
fix: install only specified packages, not extra recommended ones
37f16d4
@atharvashahane28
chore: remove comments
9ab5ab3
@atharvashahane28
fix: reduce sleep time for WB scraper
5492921
@atharvashahane28
feat: dockerignore
95a0da4
@atharvashahane28
fix: delete unused Docker pipeline script
1771e52
@atharvashahane28 atharvashahane28 self-assigned this Oct 9, 2025
@atharvashahane28 atharvashahane28 linked an issue Oct 9, 2025 that may be closed by this pull request
Scraper timeout in Docker #42
Closed
@atharvashahane28 atharvashahane28 merged commit 05cb716 into main Oct 9, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@atharvashahane28 atharvashahane28

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Scraper timeout in Docker

3 participants

@atharvashahane28 @swetha4444