Skip to content

Latest commit

 

History

History
340 lines (273 loc) · 9.79 KB

elasticdl_cloud.md

File metadata and controls

340 lines (273 loc) · 9.79 KB

ElasticDL on Public Cloud

ElasticDL is a Kubernetes-native machine learning framework. This document explains how to run an ElasticDL job on a public cloud, namely, Google Kubernetes Engine (GKE).

Configure GKE Environment

Create a Project and a Kubernetes Cluster

First, we create a new project for elasticdl in web console and a new Kubernetes cluster under this project.

We will use the project id and cluster name in next steps.

Access the Kubernetes Cluster

To access GKE in a local computer, we need to install Google Cloud SDK, which includes command-line tools like gcloud.

Luckily, Google Cloud also provides Cloud Shell with gcloud installed already. In this tutorial, we use Cloud Shell to access the Kubernetes cluster. We run the following command in Cloud Shell.

export PROJECT_ID=${your_project_id}
gcloud container clusters get-credentials cluster-1 --zone us-central1-c --project ${PROJECT_ID}

Config the Kubernetes Cluster

ElasticDL jobs require pod creation and deletion permissions. Make sure you have granted related permissions to the default or other related service accounts.

export CODE_PATH=${your_code_dir}
cd ${CODE_PATH} && git clone https://github.com/sql-machine-learning/elasticdl.git
cd ${CODE_PATH}/elasticdl
kubectl apply -f elasticdl/manifests/elasticdl-rbac.yaml

ElasticDL supports elastic scheduling, and works well the priority-based scheduling of Kubernetes. We create two customized PriorityClass in the cluster, high and low.

high.yaml

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high
value: 1000000
globalDefault: false

low.yaml

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: low
value: 1000
globalDefault: false
kubectl create -f high.yaml
kubectl create -f low.yaml

Mount a Volume for the Kubernetes Cluster

First, we create a Cloud Filestore instance in web console.

Then we follow the doc to access fileshares from the Kubernetes cluster.

In this example, we create a persistent volume claim named fileserver-claim.

Submit Job to the Kubernetes Cluster

Prepare Dataset

Step 1: We generate MNIST training and evaluation data in RecordIO format.

docker run --rm -it \
  -v $HOME/.keras:/root/.keras \
  -v $PWD:/work \
  -w /work \
  elasticdl/elasticdl:dev bash -c "/scripts/gen_dataset.sh data"

The RecordIO format dataset will generated in the data directory.

Step 2: We launch a pod which mounts the volume, and use kubectl cp command to copy MNIST dataset from local to the volume.

kubectl create -f my-pod.yaml
kubectl cp data/mnist my-pod:/data

my-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: test-pod
    image: nginx:1.7.9
    volumeMounts:
    - mountPath: /data
      name: mypvc
  volumes:
  - name: mypvc
    persistentVolumeClaim:
      claimName: fileserver-claim
      readOnly: false

Submit Job

Please refer to elasticdl_local tutorial for more details. The difference is that we have to push the image to google cloud repo.

pip install elasticdl-client

cd ${CODE_PATH}/elasticdl/model_zoo

elasticdl zoo init

elasticdl zoo build --image=gcr.io/${PROJECT_ID}/elasticdl:mnist .

elasticdl zoo push gcr.io/${PROJECT_ID}/elasticdl:mnist

We launch a training job with 2 PS pods and 4 worker pods. The master pod and PS pods are set with high priority, while worker pods are set with low priority.

elasticdl train \
  --image_name=gcr.io/${PROJECT_ID}/elasticdl:mnist \
  --model_zoo=model_zoo \
  --model_def=mnist.mnist_functional_api.custom_model \
  --training_data=/data/mnist/train \
  --validation_data=/data/mnist/test \
  --num_epochs=5 \
  --master_resource_request="cpu=2,memory=2048Mi" \
  --master_resource_limit="cpu=2,memory=2048Mi" \
  --master_pod_priority=high \
  --worker_resource_request="cpu=2,memory=2048Mi" \
  --worker_resource_limit="cpu=2,memory=2048Mi" \
  --worker_pod_priority=low \
  --ps_resource_request="cpu=2,memory=2048Mi" \
  --ps_resource_limit="cpu=2,memory=2048Mi" \
  --ps_pod_priority=high \
  --minibatch_size=64 \
  --num_minibatches_per_task=64 \
  --num_ps_pods=2 \
  --num_workers=4 \
  --evaluation_steps=200 \
  --grads_to_wait=1 \
  --job_name=test-mnist \
  --log_level=INFO \
  --image_pull_policy=Always \
  --volume="mount_path=/data,claim_name=fileserver-claim" \
  --distribution_strategy=ParameterServerStrategy

To see the status of each pod:

kubectl get pods

To see the loss in a worker pod:

kubectl logs elasticdl-test-mnist-worker-0 | grep "Loss"

To see the evaluation metrics in the master pod:

kubectl logs elasticdl-test-mnist-master | grep "Evaluation"

Example of Job Fault Tolerance

ElasticDL supports fault tolerance in distributed training. When a worker pod is killed, the training job does not crash and the master pod will try to relaunch a new worker pod.

At first, all pods are running:

elasticdl-test-mnist-master     1/1     Running   0          35s
elasticdl-test-mnist-ps-0       1/1     Running   0          29s
elasticdl-test-mnist-ps-1       1/1     Running   0          28s
elasticdl-test-mnist-worker-0   1/1     Running   0          28s
elasticdl-test-mnist-worker-1   1/1     Running   0          28s
elasticdl-test-mnist-worker-2   1/1     Running   0          28s
elasticdl-test-mnist-worker-3   1/1     Running   0          28s

Then, we delete a worker pod:

kubectl delete pod elasticdl-test-mnist-worker-0

The master pod creates a new worker pod elasticdl-test-mnist-worker-4 at once.

NAME                            READY   STATUS    RESTARTS   AGE
elasticdl-test-mnist-master     1/1     Running   0          51s
elasticdl-test-mnist-ps-0       1/1     Running   0          45s
elasticdl-test-mnist-ps-1       1/1     Running   0          44s
elasticdl-test-mnist-worker-1   1/1     Running   0          44s
elasticdl-test-mnist-worker-2   1/1     Running   0          44s
elasticdl-test-mnist-worker-3   1/1     Running   0          44s
elasticdl-test-mnist-worker-4   1/1     Running   0          6s

Example of Elastic Scheduling

After we launch the MNIST training job, we launch another nginx service with high priority in the same cluster.

nginx.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-nginx
  labels:
    app: nginx
spec:
  replicas: 5
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        imagePullPolicy: Always
        ports:
        - containerPort: 80
        resources:
          limits:
            cpu: 2
            memory: 2048Mi
            ephemeral-storage: 1024Mi
          requests:
            cpu: 2
            memory: 2048Mi
            ephemeral-storage: 1024Mi
      priorityClassName: high
      restartPolicy: Always
kubectl create -f nginx.yaml

We will find that some worker pods with low priority are preempted by nginx pods with high priority.

NAME                            READY   STATUS        RESTARTS   AGE
elasticdl-test-mnist-master     1/1     Running       0          34s
elasticdl-test-mnist-ps-0       1/1     Running       0          27s
elasticdl-test-mnist-ps-1       1/1     Running       0          27s
elasticdl-test-mnist-worker-0   1/1     Running       0          27s
elasticdl-test-mnist-worker-1   1/1     Terminating   0          27s
elasticdl-test-mnist-worker-2   1/1     Terminating   0          27s
elasticdl-test-mnist-worker-3   1/1     Terminating   0          26s
test-nginx-7585fc5976-5hs7h     1/1     Running       0          2s
test-nginx-7585fc5976-9s4nx     1/1     Running       0          2s
test-nginx-7585fc5976-bf2th     0/1     Pending       0          2s
test-nginx-7585fc5976-ckd94     0/1     Pending       0          2s
test-nginx-7585fc5976-ss8pk     0/1     Pending       0          2s

After preemption, the training job still goes on with one worker pod.

elasticdl-test-mnist-master     1/1     Running   0          61s
elasticdl-test-mnist-ps-0       1/1     Running   0          54s
elasticdl-test-mnist-ps-1       1/1     Running   0          54s
elasticdl-test-mnist-worker-0   1/1     Running   0          54s
elasticdl-test-mnist-worker-4   0/1     Pending   0          26s
elasticdl-test-mnist-worker-5   0/1     Pending   0          26s
elasticdl-test-mnist-worker-6   0/1     Pending   0          26s
test-nginx-7585fc5976-5hs7h     1/1     Running   0          29s
test-nginx-7585fc5976-9s4nx     1/1     Running   0          29s
test-nginx-7585fc5976-bf2th     1/1     Running   0          29s
test-nginx-7585fc5976-ckd94     1/1     Running   0          29s
test-nginx-7585fc5976-ss8pk     1/1     Running   0          29s

Then, we scale the nginx deployment down to 1 replica. Some cluster resources are freed.

kubectl scale deployment.v1.apps/test-nginx --replicas=1

We find that the training job takes over the freed resources, and goes on with 4 worker pods.

NAME                            READY   STATUS    RESTARTS   AGE
elasticdl-test-mnist-master     1/1     Running   0          2m3s
elasticdl-test-mnist-ps-0       1/1     Running   0          116s
elasticdl-test-mnist-ps-1       1/1     Running   0          116s
elasticdl-test-mnist-worker-0   1/1     Running   0          116s
elasticdl-test-mnist-worker-4   1/1     Running   0          88s
elasticdl-test-mnist-worker-5   1/1     Running   0          88s
elasticdl-test-mnist-worker-6   1/1     Running   0          88s
test-nginx-7585fc5976-5hs7h     1/1     Running   0          91s