This doc will show you how to build a modern distributed system from the source. I will use the Ray-project as the example.
The official document about this topic could be found at: https://ray.readthedocs.io/en/latest/installation.html#building-ray-from-source
As I followed the introduction, I have been running into many problems. This document will note down everything while building the Ray from the source. You will find some trouble shootings at the bottom of this doc.
Chose a dir, for example ~/projects, and run:
git clone https://github.com/ray-project/ray.git
Please refer to https://docs.bazel.build/versions/master/install-os-x.html. It is easy enough to read. "Installing using binary installer" is recommended.
First of all, go to App Store, download Xcode and install it.
Then install the Homebrew:
$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
Update the PATH:
export PATH="/usr/local/opt/python/libexec/bin:$PATH"
Install python, pip should be installed together automatically:
brew install python
Install the virtualenv:
pip3 install virtualenv
Create a dir for python virtual env:
mkdir pythonvenv
cd pythonvenv
Within the dir, create the venv. At the time of this writing, Ray just released 0.8.0, The source is built with python 3.6:
virtualenv --no-site-packages --python=python3.6 venv_github_ray_36
Activate the venv. BTW, the command for stopping the virtual env is "deactivate"
source ./venv_github_ray_36/bin/activate
Check if virtual env is active:
which python
The output should be something like: ~/projects/pythonvenv/venv_github_ray_36/bin/python
Install some prerequisite tools:
pip install cython==0.29.0 setuptools_scm numpy six tensorflow==1.7.1 pyhocon
Now use the same terminal and go the ray/python dir and run:
pip install -e . --verbose
If you see errors like:
ERROR: Could not find a version that satisfies the requirement tensorflow==1.7.1 (from versions: 1.13.0rc1, 1.13.0rc2, 1.13.1, 1.13.2, 1.14.0rc0, 1.14.0rc1, 1.14.0, 1.15.0rc0, 1.15.0rc1, 1.15.0rc2, 1.15.0rc3, 1.15.0, 2.0.0a0, 2.0.0b0, 2.0.0b1, 2.0.0rc0, 2.0.0rc1, 2.0.0rc2, 2.0.0, 2.1.0rc0, 2.1.0rc1, 2.1.0rc2) ERROR: No matching distribution found for tensorflow==1.7.1
Please check your python version. Maybe you are using python 3.7. Current build only supports python 3.6.
If your build is blocked while downloading the pyarrow from https://s3-us-west-2.amazonaws.com/arrow-wheels/3a11193d9530fe8ec7fdb98057f853b708f6f6ae/index.html, you might have network issue. Try download pyarrow-0.14.0.RAY-cp36-cp36m-macosx_10_6_intel.whl from the s3 page and run:
pip install -q --target="~/projects/github/ray/python/ray/pyarrow_files" pyarrow==0.14.0.RAY --find-links ~/Download/pyarrow-0.14.0.RAY-cp36-cp36m-macosx_10_6_intel.whl --upgrade
run the following script the skip download the pyarrow again while runnig the build script:
export SKIP_PYARROW_INSTALL=true
If you see something like this:
Please go to ray dir and run:
bazel build ray_pkg
This build sould be failed and you should see some output like:
WARNING: Download from https://github.com/google/boringssl/archive/83da28a68f32023fd3b95a8ae94991a07b1f6c62.zip failed: class javax.net.ssl.SSLProtocolException Read timed out
Which shows that there is a network issue. Try to use a proxy.
If you installed some dependencies on your local mac, e.g. boost, you may run into some cpp build issues. Please make sure either uninstall it or keep the installed version as same as the version used in bazel/ray_deps_setup.bzl.