Product Guide
TigerTrade is a delightful and pleasant way to buy from and sell to members of the Princeton community, as a successor to the current TigerTrade app. Our application improves upon the old system in a number of ways:
Semantic Search: Currently, TigerTrade searches for exact word matches in listings, so searching "bicycle", for example, will not match "bike". Our application not only enables this kind of searching, but also does a fuzzy searching, so "cycle" will also match "bicycle". This casts a wider net so that you are more likely to find items that matter to you.
Search Watching: If the item you're searching for is not found, quickly create an alert to notify you as soon as a matching listing appears.
Image Upload: Images are a first-class part of TigerTrade. Users can upload up to 10 images, which are displayed elegantly with their listing. This allows fashion items and other highly-visual items to be given appropriate visual, as well as texual description.
Users can simply navigate to tigertra.de to start using TigerTrade. Many features will require a login, which can be accessed after clicking the login button and signing in through CAS.
When a user first visits TigerTrade, they are presented with the listings view. In this view, users will be able to search, sort, view, and act on listings presented to them.
- Search Bar: TigerTrade will intelligently search for any listings in the app that match the user's query.
- Filter Bar: The filter bar allows the user to further filter their query and modify their view of the listings. We further discuss the filter bar below.
- Listing Photos: All photos are displayed as a horizontal gallery. Users can click on the photos to view larger versions of the thumbnails.
- Listing Actions: Any actions the user may wish to take on an action are displayed here. The permalink button is for users who wish to copy a link to the individual listing to share with others.
- Listing Creation:The user can create a listing by clicking the Floating Action Button on the bottom right. This opens a compose window, as described below.
In the expanded view of the filter bar, users can further parametrize their queries, setting whether to:
- Show only favorited posts
- Bounds on creation dates for displayed posts
- Minimum and maximum prices
- Include already-sold listings in results
Users can additionally sort the results by:
- Most Recently Created
- Oldest
- Cheapest
- Most Expensive
- Soonest Expiration
- Furthest Expiration
Users receive slightly different information and options when viewing their own posts.
-
My Listings vs Recent Listings: The navigation menu allows for users to choose between viewing only their own posts or viewing all users' posts.
-
Actions: Instead of having the option to Contact Seller, users can Edit, Delete, or Mark as Sold their own listings.
The Compose window allows the user to compose a new post or edit an existing one. The expiration date is automatically set to be one year from the current date, but the user can change that. The Compose window can be closed by clicking the "X" or minimized by clicking the top bar of the overlay.
This view allows users to create and interact with requests to buy items. For example, if a user cannot find any bicycles in the system, they can create a buy request for bicycles and have sellers contact them. Users can also automatically "watch" their buy request, which creates a watched search from the title of their buy request.
This portion functions very similarly to the listings workflow, so we will omit a more detailed walkthrough.
When users click "Watch this Search" while searching for listings, the search is saved in Watched Searches. From this view, users can view the currently-active listings that match their search, stop watching the searches, and toggle whether they wish to receive email notifications on new matches for a given watched search. While our listing search results make a "fuzzy search" that allows for matches on parts of words, email notifications will be sent only if a full-word match is found between the search query and the title and body of a new listing.
TigerTrade is implemented as a single page application with standard frontend and backend connection over a RESTful API.
We assume that you have installed and configured Git, Go 1.8 or later, and Yarn, and have access to our repository. (If you are running an Ubuntu-derived OS that uses Go 1.6 or older, you can install Go 1.8 by following the instructions on the Ubuntu wiki.) Additionally, make sure you have set the environment variable $GOPATH to a reasonable value. We recommend export GOPATH=~/go. We make no guarantees about operating systems besides macOS and Linux, as our team exclusively used those operating systems.
The following commands worked to set up on a fresh install of Ubuntu 14.04:
sudo apt install git npm curl
sudo apt-add-repository ppa:longsleep/golang-backports
curl -sL https://deb.nodesource.com/setup_7.x | sudo -E bash -
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
sudo apt update
sudo apt install yarn golang-go nodejs
echo 'export GOPATH=~/go' > ~/.bashrc
source ~/.bashrcThen to install, run:
$ go get github.com/casey-chow/tigertrade # this downloads the code into the correct directory
$ cd $GOPATH/src/github.com/casey-chow/tigertrade
$ make dev # prepare a development environment
$ make install # install all dependencies
$ cp .env.example .env; nano .env # set up environment for running serverFrom here, make sure that you have a proper .env file, either by copying and configuring .env.example or by adding one given to you. Then, in one terminal, run:
$ make serve # this starts an auto-reloading development api serverand in another terminal, run:
$ make serve-client # this starts an auto-reloading frontend serverFrom there, your browser should automatically open the main page for development.
The following commands can be run from the project root, and should prove useful to a developer:
Server
make install Install all dependencies
make build Builds the server
make docs Starts a server for docs
make serve Runs a hot-reloading server for development
make test Runs the test suite once
make test-watch Runs a pretty testing server for the API code
Client
yarn start Runs an auto-reloading dev server for the frontend
yarn build Builds and bundles the client code
yarn test Runs the test suite for the client
Both
make dev Creates a development environment
make clean Removes all temporary files
make purge Uninstalls all dependencies, removes temp files
We found in practice that one of the most effective ways of reducing code written and app configuration is to smartly employ external hosted services.
Heroku: This is used to actually host the application (both the frontend and the backend). Our Heroku configuration is set up with a staging and production app for both our staging and production servers (the staging server, with dummy data, is available at https://stage.tigertra.de). Per industry practice, secret information used is stored as environment variables in the platform. Heroku also provides us with managed Postgres databases, of which we have one for staging, and one for production.
Cloudflare: We use Cloudflare for speed optimizations and DNS--this allows us to perform over-the-wire compression, asset caching, and content distribution. Cloudflare is set to reset all its caches whenever we deploy a new version of the site to production (see "Building and Deploying").
Amazon S3: Amazon's Simple Storage Service provides a simple way to store user-uploaded images. For security and cost-cutting, we set the service to delete images after a year, which is generally more than enough time for the listing the image is attached to to be sold. Storage is incredibly cheap nowadays, and because of resizing we expect very little cost from this platform.
Smartlook: To understand usage, we knew that Google Analytics woud largely be useless for such a small user base, so we instead used Smartlook, which allows us to get a screencast of every user's usage of our site.
Sentry: Sentry is an excellent error handling tool, which is very useful for catching problems in production. We use it to track errors that may occur while running the app.
SendGrid: We use SendGrid to send our emails. Each of the emails we send through the site are done using Transactional Templates sent through the backend.
The project repository itself is architected with the following folder structure:
client/src/ client application code
server/ server code
hooks/ useful development hooks, installed using make dev
node_modules/ Javascript dependencies
vendor/ Go dependencies
We found that we are able to keep both client and server in the same repository, and thus sync any changes to them in revision control, by deploying to Heroku with different configurations for frontend and backend servers--essentially, the frontend server will ignore backend code and vice versa, allowing both of them to play together nicely.
We aim for Security by Simplicity--that is, taking simple approaches to development that make it as obvious as possible whether we have security issues. Here, we detail our defenses against common attacks on web applications.
Cross-Site Scripting: Our architecture does no server-side HTML rendering, and since React doesn't ever parse the data it receives as HTML, our site is inherently XSS-resistant as long as everything we do is rendered using React (which we believe it is).
Cross-Site Request Forgery: We prevent CSRF attacks using the Origin and referrer headers, which both easy to implement in a RESTful API and effective against these attacks. This check is implemented as a middleware in our server and so is taken care of automatically for any request the server receives. Put simply, the middleware checks whether the origin header is valid, then if the referrer header is valid, and rejects the request with a 403 Forbidden if both are invalid. (It does allow the request if there are no such headers, however. This runs against the OWASP guidelines but is useful for debugging, and at if a user's browser can be hijacked such that it wouldn't send the mandatory Origin and Referer headers, which cannot be modified by Javascript, period, we assume that the user's system is too compromised for any server-side logic to filter out invalid requests without CAPTCHAs or risk analysis.)
SQL Injection: We prevent SQL injection by using prepared statements in our SQL queries. This is handled automatically by our SQL generation library, Squirrel. At no point do we string concatenate or interpolate SQL off of user-provided values, and so there should be no vulnerability (at least in our code) to SQL injection.
Resource Overload: This is not an issue for client-side since we don't pay for the client resources and a single client being slow doesn't affect other users. On the server, since Go is an inherently fast language, and Cloudflare offers DoS protection, attempting to perform any abuse of our site to take it down would likely fail. Further, users can only add content once authenticated and their identities on the site are linked with real identities, disincentivizing such attacks.
Finally, rather than trying to secure the system against abusive use for images, we decided to set up our storage to log who uploads what image and delete images after a year. We have notifications set up if the amount stored exceeds a certain threshold, and can restrict photo uploads from there. We also reduce photo usage by resizing and compressing all photos. Additionally, we validate the filetypes of the uploaded images by magic number rather than by extension, and modify the images before storing them, so sending poisoned photos would be very difficult.
We expose a simple RESTful API for our frontend to consume. Importantly, our frontend and backend servers are separate--the frontend server simply sends bundled frontend code, while the backend server simply serves an API. This allows us to use CAS, which is whitelisted for *.herokuapp.com, without exposing an ugly Heroku subdomain as our site URL, and simplifies our backend.
This, however, means that we need to enable cross origin resource sharing, a technology used to work around the same origin policy. The URL for the server and client to communicate with one another are encoded in environment variables in their Heroku configurations, so that the client knows where to send its requests, and the server knows what client origins to expect.
Our database consists of the following tables:
- users: Every user is added to this table upon logging in. The table includes each user's netID as well as a unique keyID (to avoid collision of two users with the same username from, say, IAS and CAS).
- listings: Includes all the items listed to be sold. When items expire or are marked as sold, their status is changed to "inactive", but they are not removed from the database.
- seeks: A counterpart to the listings table, this table includes all the requests to buy. It has fewer fields than the listings table, justified by the use case.
- saved_searches: Stores information about user "watches". (They were orginally called saved searches and were renamed on the frontend.)
- starred_listings: Gerund table containing the many-to-many relationship for users favoriting listings. (It was originally called starring and was changed on the frontend.) Each row represents one user starring one listing.
Our architecture is built in Go using the default net/http package and httprouter, a high-performance router for net/http. We use Negroni for middleware, which allows us to modify the request and responses for all handlers, say, for example, to block potential CSRF attacks. On the data end, rather than relying on an ORM we make manual requests to the database using the pq driver for Postgres and the Squirrel SQL generator. (Data validation largely occurs at the database level, rather than at the application server.) For authentication, we manually forked and modified an existing CAS library for our needs.
In general, our architecture favors composition of small, stable packages rather than any monolithic framework.
The server code is organized loosely in the MVC pattern, without the views, since the frontend takes care of the view, so we can just render out the data as JSON. More applicably, every (non-helper) function in the folder represents the handler for a path that is dispatched by the router as necessary.
database.sql a dump of the database schema, used to track the database schema to the code that expects it
emails.go hooks for sending emails
helpers.go useful heleprs for controllers
listings.go CRUD operations and starring for listings
main.go entrypoint, includes initializers and middleware
photos.go http handler for uploading photos
router.go route to controller mappings
savedsearches.go CRUD operations for watched searches
seeks.go CRUD operations for seeks
users.go user login, logout, and login check
Each non-helpers file in server has a corresponding file in server/models that handles making requests to the database and web services, with the exception of photos.go, where the upload logic was simple enough as not to warrant a model.
Each of the main datatypes have some subset of the create-read-update-delete operation handlers, as defined and named below. All handlers should return HTTP-compliant response status codes, and further descriptions are available.
-
ReadDatatypes: read multiple of the datatype, potentially filtered by query parameters -
ReadDatatype: reads the specified datum -
CreateDatatype: creates the specified datum, and return it -
UpdateDatatype: updates the specified datum -
DeleteDatatype: deletes the specified datum
We try to avoid custom handlers and push for resourceful routing whenever possible to reduce the number of design decisions we need to make. Here are important custom handlers that handle key functionality:
CreatePhoto in photos.go receives a photo, resizes it (using the imaging library), uploads to S3, and returns the URL of the uploaded photo. Note that it stores the user that uploaded the photo in the object name in S3, for accountability reasons.
ContactListing and ContactSeek in emails.go sends a contact email to the owner of a listing or seek, and notifies the user of this email. The template is set using Sendgrid transactional templates.
UpdateListingStar in listings.go sets whether the user favorited a listing. This is implemented by adding or removing a row to the listings-stars gerund table.
GetCurrentUser in users.go returns with the profile of the current user. Creates the user if they do not yet exist. (This is secure because we are using CAS as our source of truth, not the user.)
RedirectUser in users.go does one of two things:
- If logged in, redirects back to the given redirect URL (sanitizing for validity and correct host origin).
- Otherwise, redirects to CAS login with the proper parameters.
LogoutUser in users.go does one of two things:
- If logged out, redirects to the root of the page.
- Otherwise, redirects to CAS logout with the proper parameters.
All middleware is defined in server/main.go. Below is a brief summary of what each middleware does:
-
casMiddleware: enables CAS authentication in handlers -
sentryMiddleware: reports any handler panics to Sentry -
logMiddleware: logs http requests to the console -
corsMiddleware: sends CORS headers as appropriate -
csrfMiddleware: blocks potential cross-site request forgery, as outlined below in "Security"
After any user creates or updates a listing, we dispatch a goroutine to check whether the listing matches any watches. (This is performant because goroutines are non-blocking, so this check happens asynchronously, and allows us to spend more time on checking without slowing down the user.) The code that does this is available in server/models/savedsearches.go in the checkNewListing function. Essentially, we check by writing a SQL query that is the inverse of what we used to write the query in the first place--so if the user, for example, searches for a minimum expiration date, we query for watches whose maximum expiration dates are less than the listing's. For every match, we send an email to the owner of the saved search alerting of the match.
Like for search alerting, we dispatch a goroutine on listing and seek creation (we'll call them "posts" from now on as a general term) and update to index the post's keywords. This process is found in server/models/search.go, with indexListing as the entrypoint.
- Get the post title and description, and concatenate them (we'll call this, as an egregious abuse of NLP terminology, the "corpus")
- Split the corpus into a set of words and drop all stop words
- For each word, add the word's synonyms from Princeton's WordNet database (this database is stored in memory, from a forked version of a Go package called wnram)
- Deduplicate the words, and add them to the listing's
keywordsfield
When searching, semantic matches are very simple--we simply add a query that checks whether the set search terms in the query and the keywords in the site intersect at all. This is about a 10ms operation in practice.
Our frontend is written using React and Redux architectures primarily, with the Material UI component library and various other supporting libraries--like our backend, we favored composition of the pieces we need over a monolithic solution. For dispatch, we made heavy use of the thunks, which is a middleware for Redux that allows for easier asynchronous state updates. Since frontend development for React's ecosystem is extremely well-documented, we tried as much as possible to adhere to convention so that new developers can rely on the established community documentation and support systems for React+Redux as well as the documenation we provide.
config configuration, provided by create-react-app
public static files that do not to be processed by Babel
scripts useful development utilities, called through yarn
src application source code
src/actions action creators, per http://redux.js.org/docs/basics/Actions.html#action-creators
src/components dumb components, per https://goo.gl/xDiV6C
src/containers smart containers, per https://goo.gl/xDiV6C
src/helpers useful helper functions
src/middleware redux middleware, per http://redux.js.org/docs/advanced/Middleware.html
src/reducers redux reducers, per http://redux.js.org/docs/basics/Reducers.html
In general, a container represents a "page" in our application, such as the listings view or the watched searches view. These are meant to be the primary holders of the state, and then pass that state as props through their child components.
Components are reusable UI elements, such as a card for displaying a listing's information. These are meant to be "dumb" -- making minimal direct use of the state, instead receiving the data they are meant to display via props passed by their parent containers.
In the Redux pattern, all of the application state is maintained in a single store. Any part of the app can call an action creator and dispatch the returned action. The actions contain data for the reducers to use to update the store. Actions are generally used for asynchronous operations (e.g. fetching listings from the backend) or updating parts of the application state that affect many different components.
In the Redux pattern, reducers receive dispatched actions and update the store according to the data in the action. Every element of state in the store has its own reducer that listens for dispatched actions and returns the new state of the store as a function of the action and the current state. In this way, all application data can be treated as immutable. The store only updates through the reducer processing dispatched actions, and everything that depends on the store is automatically updated after the store changes. This is the key benefit of the Redux pattern - data flows unidirectionally from the store down to the display components, and state is maintained functionally with no unexpected side effects.
Because every state update passes through the reducers, we are able to pass every state update through middleware to easily log state updates and handle errors. For example, we were able to write middleware that caught any instance of failure due to the user not being logged in and automatically requested a CAS login.
The entry point of our app is public/index.html, which is what is loaded when accessing the endpoint of the frontend dev server. The div with id="root" serves as the container into which the src/containers/Root.js is rendered.
The Root container does not directly handle any display elements; it applies the Material-UI theme, renders src/containers/App.js, and connects App to the Redux store.
The App container handles routing, rendering any of the following containers depending on the current URL route:
src/containers/Listing.js
src/containers/Listings.js
src/containers/Seek.js
src/containers/Seeks.js
src/containers/Watches.js
App also renders src/components/ComposeOverlay.js if the user has the Compose window open, renders Snackbars to give feedback to the user on the results of asynchronous actions, and renders src/components/FilterBar.js, which displays search filtering and sorting options when appropriate to the route.
The Listings, Seeks, and Watches containers display query results for listings, buy requests, and watched searches, respectively. The ListingsList, SeeksList, and WatchesList components display lists of ListingCard, SeekCard, and WatchCard, which are components for displaying the information of a single listing, buy request, or saved search. The Listing and Seek containers display instead a single ListingCard or SeekCard, and render permalink views.
It is useful to note that, while the user-facing name is Buy Requests, while developing the application we referred to buy requests as Seeks.
src/reducers/index.js contains a composeState reducer.
composeState contains the following fields:
-
show :: bool- whether the Compose overlay is currently visible -
isEdit :: bool- whether the Compose overlay is open for editing a post rather than composing a new one -
mode :: string- either "listings" or "seeks"; indicates the kind of post being composed -
listing :: object- the current listing being composed (undefinedif composing a buy request) -
seek :: object- the current seek being composed (undefinedif composing a listing)
The root view of the Compose workflow is in src/components/ComposeOverlay.js. Depending on composeState, the overlay renders either a ComposeForm (for composing listings) or a SeekComposeForm (for composing buy requests). We use redux-form for managing the state of the Compose form, allowing us to prepopulate the form with default values (or preexisting values in the case of editing a post) and send the user-inputted values to our server on submission.
The primary user flow of our application revolves around searching the main Listings and Seeks views, which is implemented with a cycle of actions and reducers to control the currentQuery in the Redux global store. User interactions with the search bar, filter bar, and card buttons dispatch loadListings or loadSeeks actions with a query object and optional reset parameter.
The currentQuery reducer patches every attribute of the dispatched query onto itself unless reset is set, in which case it will wipe the current query and replace it with the action's query. (Helpfully, this means that components do not need to be connected to the Redux store in order to update the currentQuery, allowing us to write much more modular "dumb" components.) For instance, when the Sort Order dropdown in the filter bar dispatches an action with a query like { order: 'creationDateDesc' }, the new query will be the same as the old query but with only the order changed. Conversely, when the user submits a new listing and is redirected to the "My Listings" page, the compose view dispatches an action with reset enabled and the query { isMine: true } to clear all of the other search parameters. In order to clear a field of the query, give undefined as the field's value: for instance, when the user clears the Created After field, the query { minCreateDate: undefined } is dispatched.
The modified query is then converted to a string by the query-string library for use as a parameter to a GET request for listings or buy requests from the backend. The backend returns an appropriate array of posts to display, and these overwrite the listings and seeks state.
Our application supports storing and retrieving the current query in the URL. Users can bookmark a link to the site and reopen that link to find the appropriate search results for their query. The parseQuery helper function parses the current URL route into a JavaScript object that is used to initialize the currentQuery in the store, parsing strings appropriately. Whenever the query is modified, the writeHistory helper method is used to update the route to reflect the current query.
To introduce a new query parameter on the frontend, one must:
- create a graphical element to allow the user to control this parameter and connect it
- add default values
parseQueryandstripQueryin the fileclient/src/helpers/query.jsto control when the parameter should be included in and retrieved from the URL - add backend support for parsing this parameter from the request URL
The currentQuery object has many of the same attributes as a Watched Search: this is no coincidence. Both the "Watch this Search" button on the Listings view and the "Notify Me" button on an individual buy request can be used to create a Watched Search, which is implemented by a postWatch action taking the currentQuery or seek as appropriate. Then, the "View Results" button just takes the watch object and issues a reset-ing LOAD_LISTINGS_REQUEST with that object as the query.