S. Felix Wu, wu@cs.ucdavis.edu
Fredrik Erlandsson, fredrik.erlandsson@bth.se
This crawler consists of two parts, the agent.php that does the actual crawling and a controller (found in contoller/) keeping track of the current crawling status.
The agent is dependent on the Facebook PHP SDK. To install just do a submodule update:
git submodule update --init
Most of the time you only need to use the agent.
Create a Facebook application at: https://developers.facebook.com/apps, make sure to fill in offline_access & read_stream under Permissions->Extended Permissions.
Copy config/config-dist.php to config/config.php and fill APPID, APPSEC (from your Facebook application page) & the URL to a running controller.
run php agent.php token=FACEBOOK_USER_TOKEN
or as a web application http://example.com/agent.php?token=FACEBOOK_USER_TOKEN
To run multiple instances (reccomended) of the agent in one environment use
the script bgxgrp.sh as:
bash bgxgrp.sh <#-instances> php agent.php token=FACEBOOK_USER_TOKEN
where <#-instances> should be replaced with the number of threads to run
(something between 8-15 is reasonable to not hit Facebook's 600/600 limit).
The FACEBOOK_USER_TOKEN is generated via the graph explorer page https://developers.facebook.com/tools/explorer/ using an user that is said to be over 18 of age to support crawling of all types of pages.
Happy crawling!!