Skip to content

Adds support for deploying PySpark models #196

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Corey-Zumar merged 22 commits into from
Jun 5, 2017
Merged

Adds support for deploying PySpark models #196

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
8ebebed
saving pyspark looks like it works
dcrankshaw Jun 3, 2017
2b83074
more work on pyspark container. untested
dcrankshaw Jun 3, 2017
d3fc6c7
updated docker files
dcrankshaw Jun 3, 2017
f45fa0f
fixed imports
dcrankshaw Jun 3, 2017
e5a77b2
initialization of pyspark docker container works
dcrankshaw Jun 3, 2017
58090d8
pyspark deploy integration tests pass
dcrankshaw Jun 3, 2017
8a21741
got mllib tests passing again
dcrankshaw Jun 4, 2017
ff31db3
pyspark deploy is working
dcrankshaw Jun 4, 2017
63cbf9a
download spark if not present
dcrankshaw Jun 4, 2017
74cdb85
format code
dcrankshaw Jun 4, 2017
a84d2c4
format code
dcrankshaw Jun 4, 2017
5f7da90
fixed name of pyspark dockerfile
dcrankshaw Jun 4, 2017
0fd8a00
cleanup
dcrankshaw Jun 4, 2017
1b1920d
moved failing clipper_manager test to end
dcrankshaw Jun 4, 2017
83e7caf
added data dependence
dcrankshaw Jun 4, 2017
5b3ddc7
format code
dcrankshaw Jun 4, 2017
8e4fb0c
Fixed tests
dcrankshaw Jun 4, 2017
7a63115
merged develop
dcrankshaw Jun 5, 2017
874e042
addressed review comments.
dcrankshaw Jun 5, 2017
3a3815f
fix labels issue
dcrankshaw Jun 5, 2017
b885989
fixed deploy_pyspark labels
dcrankshaw Jun 5, 2017
c9331c3
fixed label arg order in deploy_pyspark_model
dcrankshaw Jun 5, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions PySparkContainerDockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
FROM clipper/py-rpc:latest

MAINTAINER Dan Crankshaw <dscrankshaw@gmail.com>

COPY containers/python/python_container_conda_deps.txt /lib/

RUN curl -o /spark.tgz https://d3kbcqa49mib13.cloudfront.net/spark-2.1.1-bin-hadoop2.7.tgz \
&& cd / && tar zxf /spark.tgz && mv /spark-2.1.1-bin-hadoop2.7 /spark \
Copy link
Contributor

@feynmanliang feynmanliang Jun 4, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we extract vars for these versions and envsubst or BUILDARGS them in?

&& echo deb http://ftp.de.debian.org/debian jessie-backports main >> /etc/apt/sources.list \
&& apt-get update --fix-missing \
&& apt-get install -yqq -t jessie-backports openjdk-8-jdk \
&& conda install -y --file /lib/python_container_conda_deps.txt \
&& pip install findspark

COPY containers/python/pyspark_container.py containers/python/pyspark_container_entry.sh /container/
COPY clipper_admin/ /lib/clipper_admin/

ENV SPARK_HOME="/spark"

CMD ["/container/pyspark_container_entry.sh"]

# vim: set filetype=dockerfile:
Copy link
Contributor

@Corey-Zumar Corey-Zumar Jun 5, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove commented code

Copy link
Contributor Author

@dcrankshaw dcrankshaw Jun 5, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This actually isn't a comment, it's a vim modeline directive http://vim.wikia.com/wiki/Modeline_magic

5 changes: 3 additions & 2 deletions PythonContainerDockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,12 @@ FROM clipper/py-rpc:latest

MAINTAINER Dan Crankshaw <dscrankshaw@gmail.com>

COPY containers/python/python_container.py containers/python/python_container_entry.sh /container/
COPY containers/python/python_container_conda_deps.txt /lib/
RUN conda install -y --file /lib/python_container_conda_deps.txt

COPY containers/python/python_container.py containers/python/python_container_entry.sh /container/
COPY clipper_admin/ /lib/clipper_admin/

RUN conda install -y --file /lib/python_container_conda_deps.txt

CMD ["/container/python_container_entry.sh"]

Expand Down
1 change: 1 addition & 0 deletions bin/build_docker_images.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ cd $DIR/..
docker build -t clipper/py-rpc -f ./RPCDockerfile ./
time docker build -t clipper/noop-container -f ./NoopDockerfile ./
time docker build -t clipper/python-container -f ./PythonContainerDockerfile ./
time docker build -t clipper/pyspark-container -f ./PySparkContainerDockerfile ./
time docker build -t clipper/sklearn_cifar_container -f ./SklearnCifarDockerfile ./
time docker build -t clipper/tf_cifar_container -f ./TensorFlowCifarDockerfile ./
cd -
14 changes: 13 additions & 1 deletion bin/run_unittests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,18 @@ function run_frontend_tests {
function run_integration_tests {
echo -e "\nRunning integration tests\n\n"
cd $DIR
if [ -z ${SPARK_HOME+x} ]; then
echo "Downloading Spark"
curl -o spark.tgz https://d3kbcqa49mib13.cloudfront.net/spark-2.1.1-bin-hadoop2.7.tgz
Copy link
Contributor

@feynmanliang feynmanliang Jun 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we pull vars for these versions?

tar zxf spark.tgz && mv spark-2.1.1-bin-hadoop2.7 spark
export SPARK_HOME=`pwd`/spark
else
echo "Found Spark at $SPARK_HOME"
fi
pip install findspark
python ../integration-tests/clipper_manager_tests.py
python ../integration-tests/deploy_pyspark_models.py
python ../integration-tests/deploy_pyspark_pipeline_models.py
python ../integration-tests/many_apps_many_models.py 2 3
}

Expand All @@ -131,11 +142,12 @@ function run_all_tests {
redis-cli -p $REDIS_PORT "flushall"
run_management_tests
redis-cli -p $REDIS_PORT "flushall"
run_integration_tests
redis-cli -p $REDIS_PORT "flushall"
run_jvm_container_tests
redis-cli -p $REDIS_PORT "flushall"
run_rpc_container_tests
redis-cli -p $REDIS_PORT "flushall"
run_integration_tests
}

if [ "$#" == 0 ]
Expand Down
Loading