Skip to content

Add automatic scraping#17

Open
nazime1 wants to merge 18 commits intomainfrom
nazime1
Open

Add automatic scraping#17
nazime1 wants to merge 18 commits intomainfrom
nazime1

Conversation

@nazime1
Copy link

@nazime1 nazime1 commented Feb 18, 2022

python-app.yml will run every Python file in the scraper directory and push any files that said Python files create to the scraper directory.

Erica Nazimowitz and others added 18 commits February 11, 2022 16:32
Very basic scraper for practice
This one reads a bunch of inspirational quotes and creates a table to be saved into a .csv file
* Move documentation into its own directory

* Update documentation, fixes #8.

* Fix misaligned table for the section properties

* Reorganize scraper repository (#12)

* Delete old scrapers

* Move documentation into a separate directory

* Remove old json files from gitignore

* Add some basic goals to the README

* Basic start of Clojure course catalog scraper

* Clean up Clojure, it was a fun experiment while it lasted

* Merge sis.rpi.edu/course_scrapper.py into lily-rewrite

* Rename course scraper script

* Delete data/ directory

* Create python-app.yml

Testing GitHub Actions with auto-executing Python files as soon as they're published to the repository

Co-authored-by: Lily Ritter <lily@lilychan.dev>
Co-authored-by: Trevor Brunette <trevorbrunette@gmail.com>
Co-authored-by: Lily Ritter <70179370+lilyritt@users.noreply.github.com>
Should have fixed the error with not scraping in the wrong place, also added working directory in order to upload to the data repository
Trying again with pushing to data repository
@ghost
Copy link

ghost commented Mar 25, 2022

So we currently have a lil problem that running this action will accidentally push the src/ directory rather than the data/ directory (see the attached screenshot). I'm going to make a test repository for you to work with in the meantime. So please push to https://github.com/rpi-crisis/data-test while you are still testing.

Thanks for the contributions!

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants