Skip to content

Setup

Jump to bottom
Jessica Zhu edited this page Feb 19, 2021 · 12 revisions

You can run Convo and its services all separately by running each server in a separate terminal or by using Docker. If you run into any errors during the setup process, please first try consulting the Common Errors page.

Clone the Repository

The first thing you need to do is clone the Convo repository.

Setup Database

Before running the Convo servers, you need to create a database for storing user information and programs. You can do so with the following steps

  1. Go to backend/server directory in the Convo repository you cloned, and install all required Python packages using pip 
cd backend/server
pip install -r requirements.txt
  1. Start a Python shell using python3 (or just python, if it's aliased to Python 3), and in the shell environment and execute the lines
from db_manage import db
db.create_all()
  1. Exit the Python environment (with exit()) and go to the backend/server/db directory. You should see a convo.db file inside. This is the file containing the SQL database for Convo. If you have sqlite3 installed, you can access the file and query through it.

Setup Speech-To-Text

To be able to use Google's Speech-to-Text API, you must create an account and a project on Google Cloud Platform (GCP). Go to Google's Quickstart for instructions on setting everything up and getting a private key. Download your private key as a JSON file and name it gcpkey.json (or another name that you can remember).

Once you obtain your private key, move your key gcpkey.json to api. Copy the file .env.sample from api and rename it .env. Set the value of the environment variable GOOGLE_APPLICATION_CREDENTIALS in .env to the relative path of the key. For example, if you named the key gcpkey.json, the contents of the .env file should be

GOOGLE_APPLICATION_CREDENTIALS="gcpkey.json"

If you run into issues with this method, we recommend creating a service account, following the instructions here. Like before, download your service account key as a JSON file and name it servicekey.json (or another name that you can remember). Then, move servicekey.json to the api directory and the contents of the .env file should be

GOOGLE_APPLICATION_CREDENTIALS="servicekey.json"

Install the Node Modules

Currently, the Node modules won't automatically install (even with Docker), so you will have to manually do this. Before you can do so, ensure you have Node.js and npm installed on your machine. Next, go to the frontend directory and run

npm install

Setup Rasa

For the unconstrained NL to work, you have to train a model. To do so:

  1. Enter the Docker container for Rasa:
# In a new terminal, go to the convo directory and start docker:
# (You'll see a warning, "UserWarning: Could not load local model in 'models'.", but that's okay)
cd convo
docker-compose -f docker-compose.local.yml up

# In a different terminal, find the Rasa container's ID:
docker container list
# Take note of CONTAINER ID next to the "convo_rasa" IMAGE
# Now, using that CONTAINER ID, run the following command to enter the container:
docker exec -it RASA_CONTAINER_ID /bin/bash # where RASA_CONTAINER_ID is something like 4c0d64d7e54e
  1. Train the NLU model:
# Within the rasa container (in the /app directory — you can check by running `pwd`), train the model:
rasa train

Once training is completed, you should see a message like NLU model training completed. Your Rasa model is trained and saved at '/app/models/nlu-20210202-162725.tar.gz'.

  1. Exit the Docker container with the exit command (literally, exit) and close this terminal.

  2. Stop the Docker containers by going back to the first terminal (where you ran docker-compose) and hitting Control+C. You should now have a model in the models/ directory:

ls backend/rasa/models        # if you run this command...
nlu-20210202-162725.tar.gz    # you should see something like this

Note about Working Locally

On the master branch, the pages served by the UI connect via WebSockets to the Node and Python servers on your local computer by default. On different branches or older commits, the WebSockets may connect to the production servers on userstudy.appinventor.mit.edu by default (and in this case, if you make any changes to the servers on your local computer, these changes won't be reflected when you run docker-compose).

To ensure the UI is connecting to your local Node and Python servers, check that the following files have the lines with http://localhost:5000 uncommented and the lines with https://... commented out.

  1. In frontend/public/js/experiment.js:
    • Ensure line 2 (const server = 'https://userstudy.appinventor.mit.edu/api';) is commented out
    • Ensure line 4 (const server = 'http://localhost:5000';) is uncommented
  2. In frontend/public/js/socket.js:
    • Ensure lines 2-3 (const socket = io.connect("https://... and const socketApi = io.connect("https:...) are commented out
    • Ensure lines 5-6 (const socket = io.connect('http://localhost:8080'); and const socketApi = io.connect('http://localhost:5000');) are uncommented

Note about Setting Up Production Servers

On the master branch, the pages served by the UI connect via WebSockets to the Node and Python servers on your local computer by default. For running production, they should connect to the production servers.

To ensure the UI is connecting to your production Node and Python servers, check that the following files have the lines with http://localhost:5000 commented out and the lines with https://... uncommented.

  1. In frontend/public/js/experiment.js:
    • Ensure line 2 (const server = 'https://userstudy.appinventor.mit.edu/api';) is uncommented
    • Ensure line 4 (const server = 'http://localhost:5000';) is commented out
  2. In frontend/public/js/socket.js:
    • Ensure lines 2-3 (const socket = io.connect("https://... and const socketApi = io.connect("https:...) are uncommented
    • Ensure lines 5-6 (const socket = io.connect('http://localhost:8080'); and const socketApi = io.connect('http://localhost:5000');) are commented out

Setup With Docker

Docker is the recommended setup for local and remote development. In addition, the production servers are run using Docker.

To setup with Docker, make sure you have Docker and Docker Compose installed on the machine. To install both, install Docker for Mac or Docker for Windows and Docker Compose should be included. (Note that Docker runs natively on Mac and Linux systems, but may not run on Windows machines, unless you have the most up-to-date version of Windows, which you can find on the official Microsoft website. Once your Windows is up to date, try following the instructions here to set up Docker for Windows. If this doesn't work, you can also consider partitioning your machine and installing Docker for Linux.)

Once you have Docker, you should be able to run

docker -v && docker-compose -v

and output should be similar to

Docker version 19.03.2, build 6a30dfc
docker-compose version 1.24.1, build 4667896b

Docker Compose automatically runs the web, server and rasa services that are defined in the docker-compose*.yml.

  1. server - Convo Core or the backend server
  2. web - Convo VUI or frontend that allows clients to interface with Convo Core
  3. rasa - Rasa server containing a trained model

There are actually three YAML files that can be used with docker-compose - docker-compose.yml and docker-compose.prod.yml and docker-compose.local.yml

  • docker-compose.prod.yml is used in the case where an admin username and password is required to enter the website
  • docker-compose.local.yml is used for local development - main difference is no service is needed for a reverse-proxy and CertBot since everything is local

For example, to run Docker on your local machine, you can run

docker-compose -f docker-compose.local.yml up --build

(If you have previously built the docker container, you can just run, docker-compose -f docker-compose.local.yml up.)

If you are running Docker in the background (with the flag -d), to stop the system, use

docker-compose down

Once the Docker container is running, skip down to the After Setting Up section.

Setup without Docker

We recommend you setup with Docker, but if you would rather run each server manually, you can follow these instructions. You need to run three (one optional) servers to get Convo fully working.

Node Server (Frontend)

Install Node through a package manager(recommended) or through an installer. Run

npm install

in the frontend directory to install the necessary modules and lastly run

npm start

to start the Node server.

Python Server (Backend)

Install Python 3.7, if not on machine. If on MacOS, I recommend installing through Homebrew

brew install python

In the backend/server directory, install the necessary packages

cd backend/server
pip install -r requirements.txt

Start the server using

gunicorn --worker-class eventlet -w 1 --bind 0.0.0.0:5000 --log-level=info manage:app

Rasa Server

Go to the Rasa wiki page for information on Rasa and its setup.

After Setting Up

After setting up and starting the servers (running either separately or through Docker), head to http://localhost:8080 (or http(s)://[host]:8080 for the remote server). You should see the user study website.

You can head to /debug (e.g., http://localhost:8080/debug) if you want to test the Convo UI and backend server. You may also want to head to one of the user study stages (e.g., the practice stage or voice and text sandbox) with both voice and text enabled to test the voice-based system too.

You can see urls for other pages in the frontend/app.js file. For example,

app.get('/voice-and-text', (req, res) => res.sendFile(path.resolve('public/html/experiment-templates/voice-and-text.html')));

corresponds to http://localhost:8080/voice-and-text.

Clone this wiki locally