OlinSnapshot

Takes emails from the HelpMe and CarpeDiem email lists at Olin College, scrapes and sorts them, and presents them in an interactive UI

You can find the results at our main page.

To view data from Postgres Database, go to the data page.

Contributors

Mackenzie Frackleton, Kaitlyn Keil, Isa Blancett, and Wilson Tang

Contributions

Aloverso's heartbot at: https://github.com/aloverso/heartbot

Requirements

• Python3
• Virtual Environment
• Heroku CLI
• Flask
• Pip
• psycopg2
• postgresql
• Linux or OSX? (Untested in Windows)

Installing Virtual Environment, Heroku CLI, & Postgresql

Virtual Environment

python3 -m pip install --user virtualenv

Heroku CLI

For Linux:

wget -qO- https://cli-assets.heroku.com/install-ubuntu.sh | sh

For OSX:

brew install heroku/brew/heroku

Postgresql

For Linux:

sudo apt-get update
sudo apt-get install postgresql postgresql-contrib

For OSX:

brew install postgresql

Setting up Outlook & Gmail for POP

We are using a Gmail account that is forwarded emails from an Outlook account. How you set up this account may be specific to the clients you are using. If you are using Gmail, make sure that 'POP' and 'allow access from less secure apps' are both enabled. Please email isabel.blancett@students.olin.edu if you have any questions regarding this step.

On your Gmail account:

1. Settings>Forwarding and POP/IMAP>Enable POP for all mail
2. My Account>Apps with account access(under Sign-in & Security)>Allow Less Secure Apps

Getting Started

To clone and setup virtual environment:

git clone https://github.com/KaitlynKeil/OlinSnapshot.git
cd OlinSnapshot
virtualenv -p python3 venv
source venv/bin/activate
pip install -r requirements.txt

Create a heroku app heroku create appname and run git remote heroku git:remote -a <app name> to link. Make sure to give your app a unique name, because 'olin-snapshot' is taken by us!

Initializing Database

In order to set up a database in Heroku, use heroku addons to make sure it does not already exist. If heroku-postgresql appears in the app's list of add-ons, you can move on. Otherwise, you can provision it with the command heroku addons:create heroku-postgresql:<PLAN_NAME>. hobby-dev as a plan should be enough. Use heroku config to make sure you have a DATABASE_URL variable. If more help is needed, you can read about provisioning heroku postgres at this link.

Once you have done this, set up the database by running

python3 -m app.set_up_database

and create an environment variable DATABASE_URL by running

export DATABASE_URL=<copy and paste the value from heroku config here>

Once this has been created, run python3 app/set_up_database.py in order to initialize your database. This will create a schema called 'emails' with three tables: -msg: contains the email information, such as the subject (name), body, event_place (location if specified), value (integer that defaults to 5), who (sender), and msg_id (unique serial) -cats: contains category names (Food, Event, Lost, and Other) and their corresponding cat_id (serial) -msg_to_cat: join table that contains two columns. msg_id corresponds to the email serial. cat_id corresponds to the category. For each email, there is an entry for each category. Thus, if we are looking at email 5, which is both Food (cat_id 1) and Lost (cat_id 3), msg_to_cat would have two rows: 5 1 and 5 3.

The database should now be set up.

Running locally

To run simple flask app on local host:

heroku config -a <app name>
export DATABASE_URL=<value from heroku config DATABASE_URL>
export SNAPSHOT_EMAIL=<your gmail>
export SNAPSHOT_PASS=<your password>

export FLASK_APP=olinsnapshot.py
flask run

Deployment to Heroku

heroku config:set SNAPSHOT_EMAIL=<your gmail> SNAPSHOT_PASS=<your pass>
git add .
git commit -am "Initializing heroku app"
git push heroku master
heroku open

Deployed to: https://appname.herokuapp.com/

General Set Up

Most of the programs are contained in 'app'. When this is called either by heroku or by a local server, it runs __init__.py, which in turn imports routes.py. This sets up the two available pages as they are accessed: /index, which is imported from templates/index.html and displays the D3 graphics, and /data, which updates and reads the database.

When /data is accessed, routes.py uses email_scraper.py to check for new emails from the linked gmail account via poplib and connects to the database using config.py. If any emails are found, they are added to the database through postgres_parser.py. After that, postgres_parser.py loads all of the emails into any relevant categories and bundles it into a json string, which is sent to the /data page. From here, it can be read and used by D3.

Issues

• D3 cannot currently handle user events (clicks)
• Email scraper grabs Outlook account as original sender of the email
• Email content is littered with formatting
• No iCal support