This page was generated from examples/istio/canary_update/canary.ipynb.

Canary Deployment with Seldon and Istio

Prerequistes

You will need - Git clone of Seldon Core - A running Kubernetes cluster with kubectl authenticated - seldon-core Python package (pip install seldon-core>=0.2.6.1) - Helm client

Creating a Kubernetes Cluster

Follow the Kubernetes documentation to create a cluster.

*This demo needs egress when running the load test to allow MNIST digits to be downloaded. If you want to run the load test then you will need to follow the docs on egress `here <https://istio.io/docs/tasks/traffic-management/egress/#calling-external-services-directly>`__ if you run istio in a way that egress is blocked*

Once created ensure kubectl is authenticated against the running cluster.

Setup

[14]:
!kubectl create namespace seldon
namespace/seldon created
[15]:
!kubectl config set-context $(kubectl config current-context) --namespace=seldon
Context "minikube" modified.
[16]:
!kubectl create clusterrolebinding kube-system-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
clusterrolebinding.rbac.authorization.k8s.io/kube-system-cluster-admin created

Install Helm

[17]:
!kubectl -n kube-system create sa tiller
!kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
!helm init --service-account tiller
serviceaccount/tiller created
clusterrolebinding.rbac.authorization.k8s.io/tiller created
$HELM_HOME has been configured at /home/clive/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!
[18]:
!kubectl rollout status deploy/tiller-deploy -n kube-system
Waiting for deployment "tiller-deploy" rollout to finish: 0 of 1 updated replicas are available...
deployment "tiller-deploy" successfully rolled out

Setup Istio

Ensure you have istio installed. Follow their docs

For this example we will create the default istio gateway for seldon which needs to be called seldon-gateway. You can supply your own gateway by adding to your SeldonDeployments resources the annotation seldon.io/istio-gateway with values the name of your istio gateway.

Create a gateway for our istio-ingress

[19]:
!kubectl create -f ../../../notebooks/resources/seldon-gateway.yaml
gateway.networking.istio.io/seldon-gateway created

Label our namespace so istio creates sidecars

[20]:
!kubectl label namespace seldon istio-injection=enabled
namespace/seldon labeled

If you are using Minikube for your Kubernetes cluster you will need to run as root in a separte terminal:

minikube tunnel

This will allow a LoadBalancer to be simulated on your local machine.

[21]:
INGRESS_HOST=!kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
INGRESS_PORT=!kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].port}'
ISTIO_GATEWAY=INGRESS_HOST[0]+":"+INGRESS_PORT[0]

Check the istio gateway address

[22]:
ISTIO_GATEWAY
[22]:
'10.109.199.85:80'

To view the istio traffic you can go to the istio grafana dashboard. In a separate terminal port-forward to it:

See their docs here

Start seldon-core

[23]:
!helm install ../../../helm-charts/seldon-core-operator --name seldon-core --set istio.enabled=true --set usageMetrics.enabled=true --namespace seldon-system
NAME:   seldon-core
LAST DEPLOYED: Sun Jun 30 11:38:18 2019
NAMESPACE: seldon-system
STATUS: DEPLOYED

RESOURCES:
==> v1/ClusterRole
NAME                          AGE
seldon-operator-manager-role  0s

==> v1/ClusterRoleBinding
NAME                                 AGE
seldon-operator-manager-rolebinding  0s

==> v1/ConfigMap
NAME                     DATA  AGE
seldon-spartakus-config  3     1s

==> v1/Pod(related)
NAME                                         READY  STATUS             RESTARTS  AGE
seldon-operator-controller-manager-0         0/1    ContainerCreating  0         0s
seldon-spartakus-volunteer-5866b6df59-sslhd  0/1    ContainerCreating  0         0s

==> v1/Secret
NAME                                   TYPE    DATA  AGE
seldon-operator-webhook-server-secret  Opaque  0     1s

==> v1/Service
NAME                                        TYPE       CLUSTER-IP    EXTERNAL-IP  PORT(S)  AGE
seldon-operator-controller-manager-service  ClusterIP  10.96.98.198  <none>       443/TCP  0s

==> v1/ServiceAccount
NAME                              SECRETS  AGE
seldon-core-seldon-core-operator  1        1s
seldon-spartakus-volunteer        1        1s

==> v1/StatefulSet
NAME                                READY  AGE
seldon-operator-controller-manager  0/1    0s

==> v1beta1/ClusterRole
NAME                        AGE
seldon-spartakus-volunteer  0s

==> v1beta1/ClusterRoleBinding
NAME                        AGE
seldon-spartakus-volunteer  0s

==> v1beta1/CustomResourceDefinition
NAME                                         AGE
seldondeployments.machinelearning.seldon.io  1s

==> v1beta1/Deployment
NAME                        READY  UP-TO-DATE  AVAILABLE  AGE
seldon-spartakus-volunteer  0/1    1           0          0s


NOTES:
NOTES: TODO


[24]:
!kubectl rollout status statefulset.apps/seldon-operator-controller-manager -n seldon-system
Waiting for 1 pods to be ready...
partitioned roll out complete: 1 new pods have been updated...

Serve Single Model

[25]:
from random import randint,random
import json
from matplotlib import pyplot as plt
import numpy as np
from tensorflow.examples.tutorials.mnist import input_data
from seldon_core.seldon_client import SeldonClient
import seldon_core

def gen_image(arr):
    two_d = (np.reshape(arr, (28, 28)) * 255).astype(np.uint8)
    plt.imshow(two_d,cmap=plt.cm.gray_r, interpolation='nearest')
    return plt

def download_mnist():
    return input_data.read_data_sets("MNIST_data/", one_hot = True)

def predict_rest_mnist(mnist,deployment_name,namespace,istio_gateway):
    sc = SeldonClient(deployment_name=deployment_name,namespace=namespace,gateway_endpoint=istio_gateway)
    batch_xs, batch_ys = mnist.train.next_batch(1)
    chosen=0
    gen_image(batch_xs[chosen]).show()
    data = batch_xs[chosen].reshape((1,784))
    features = ["X"+str(i+1) for i in range (0,784)]
    r = sc.predict(gateway="istio",transport="rest",shape=(1,784),data=data,payload_type='ndarray',names=features)
    predictions = seldon_core.utils.seldon_message_to_json(r.response)
    print(predictions)
    #print("Route:"+json.dumps(predictions["meta"]["routing"],indent=2))
    fpreds = [ '%.2f' % elem for elem in predictions["data"]["ndarray"][0] ]
    m = dict(zip(predictions["data"]["names"],fpreds))
    print(json.dumps(m,indent=2))

[26]:
%matplotlib inline
from visualizer import get_graph
mnist = download_mnist()
Extracting MNIST_data/train-images-idx3-ubyte.gz
Extracting MNIST_data/train-labels-idx1-ubyte.gz
Extracting MNIST_data/t10k-images-idx3-ubyte.gz
Extracting MNIST_data/t10k-labels-idx1-ubyte.gz
[27]:
!kubectl apply -f mnist_v1.json
seldondeployment.machinelearning.seldon.io/mnist-classifier created
[28]:
!kubectl rollout status deploy/mnist-deployment-sk-mnist-predictor-73d7608
deployment "mnist-deployment-sk-mnist-predictor-73d7608" successfully rolled out
[29]:
predict_rest_mnist(mnist,"mnist-classifier","seldon",ISTIO_GATEWAY)
../_images/examples_istio_canary_28_0.png
{'meta': {'puid': 'nn5g30lmmgj5sqg1etdcbi4695', 'requestPath': {'sk-mnist-classifier': 'seldonio/sk-example-mnist:0.2'}}, 'data': {'names': ['class:0', 'class:1', 'class:2', 'class:3', 'class:4', 'class:5', 'class:6', 'class:7', 'class:8', 'class:9'], 'ndarray': [[0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]]}}
{
  "class:0": "0.00",
  "class:1": "1.00",
  "class:2": "0.00",
  "class:3": "0.00",
  "class:4": "0.00",
  "class:5": "0.00",
  "class:6": "0.00",
  "class:7": "0.00",
  "class:8": "0.00",
  "class:9": "0.00"
}

Start a Load Test

[34]:
!kubectl label nodes $(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') role=locust
node/minikube labeled
[44]:
!helm install ../../../helm-charts/seldon-core-loadtesting --name loadtest  \
    --namespace seldon \
    --repo https://storage.googleapis.com/seldon-charts \
    --set locust.script=mnist_rest_locust.py \
    --set locust.host=http://{ISTIO_GATEWAY} \
    --set rest.pathPrefix=/seldon/seldon/mnist-classifier \
    --set oauth.enabled=false \
    --set oauth.key=oauth-key \
    --set oauth.secret=oauth-secret \
    --set locust.hatchRate=1 \
    --set locust.clients=1 \
    --set loadtest.sendFeedback=1 \
    --set locust.minWait=0 \
    --set locust.maxWait=0 \
    --set replicaCount=1 \
    --set data.size=784

NAME:   loadtest
LAST DEPLOYED: Sun Jun 30 16:23:57 2019
NAMESPACE: seldon
STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME                   READY  STATUS       RESTARTS  AGE
locust-master-1-mblc6  0/2    Init:0/1     0         1s
locust-master-1-trgc5  0/2    Terminating  0         85s
locust-slave-1-jgqp2   0/2    Init:0/1     0         1s
locust-slave-1-mdgxc   0/2    Terminating  0         85s

==> v1/ReplicationController
NAME             DESIRED  CURRENT  READY  AGE
locust-master-1  1        1        0      1s
locust-slave-1   1        1        0      1s

==> v1/Service
NAME             TYPE      CLUSTER-IP     EXTERNAL-IP  PORT(S)                                       AGE
locust-master-1  NodePort  10.98.215.152  <none>       5557:31305/TCP,5558:31110/TCP,8089:31793/TCP  1s


Now we will add a canary and split traffic 75% to 25% to it. This is done by adding a new predictor to the SeldonDeployment and specifying the traffic values.

[40]:
!pygmentize mnist_v2.json
{
    "apiVersion": "machinelearning.seldon.io/v1alpha2",
    "kind": "SeldonDeployment",
    "metadata": {
        "labels": {
            "app": "seldon"
        },
        "name": "mnist-classifier"
    },
    "spec": {
        "annotations": {
            "project_name": "Mnist classification"
        },
        "name": "mnist-deployment",
        "predictors": [
            {
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/sk-example-mnist:0.2",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "sk-mnist-classifier",
                                "resources": {
                                    "requests": {
                                        "memory": "1Mi"
                                    }
                                }
                            }
                        ],
                        "terminationGracePeriodSeconds": 1
                    }
                }],
                "graph": {
                    "children": [],
                    "name": "sk-mnist-classifier",
                    "endpoint": {
                        "type" : "REST"
                    },
                    "type": "MODEL"
                },
                "name": "sk-mnist-predictor",
                "replicas": 1,
                "labels":{
                    "version":"v1"
                },
                "traffic": 75
            },
            {
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/tf-example-mnist:0.1",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "tf-mnist-classifier",
                                "resources": {
                                    "requests": {
                                        "memory": "1Mi"
                                    }
                                }
                            }
                        ],
                        "terminationGracePeriodSeconds": 1
                    }
                }],
                "graph": {
                    "children": [],
                    "name": "tf-mnist-classifier",
                    "endpoint": {
                        "type" : "REST"
                    },
                    "type": "MODEL"
                },
                "name": "tf-mnist-predictor",
                "replicas": 1,
                "labels":{
                    "version":"v2"
                },
                "traffic": 25
            }

        ]
    }
}
[41]:
get_graph("mnist_v2.json")
[41]:
../_images/examples_istio_canary_34_0.svg
[42]:
!kubectl apply -f mnist_v2.json
seldondeployment.machinelearning.seldon.io/mnist-classifier configured
[45]:
predict_rest_mnist(mnist,"mnist-classifier","seldon",ISTIO_GATEWAY)
../_images/examples_istio_canary_36_0.png
{'meta': {'puid': 'pii2cg6n9gpke7gag6etkuj4cv', 'requestPath': {'sk-mnist-classifier': 'seldonio/sk-example-mnist:0.2'}}, 'data': {'names': ['class:0', 'class:1', 'class:2', 'class:3', 'class:4', 'class:5', 'class:6', 'class:7', 'class:8', 'class:9'], 'ndarray': [[0.0, 0.0, 0.03333333333333333, 0.8666666666666667, 0.0, 0.06666666666666667, 0.0, 0.0, 0.03333333333333333, 0.0]]}}
{
  "class:0": "0.00",
  "class:1": "0.00",
  "class:2": "0.03",
  "class:3": "0.87",
  "class:4": "0.00",
  "class:5": "0.07",
  "class:6": "0.00",
  "class:7": "0.00",
  "class:8": "0.03",
  "class:9": "0.00"
}

You should see traffic being split on the Istio service dashboard for the mnist-classifier.

skpredictor

tfpredictor

When you are happy the canary is ok you can promote to full traffic.

[46]:
!kubectl apply -f mnist_v3.json
seldondeployment.machinelearning.seldon.io/mnist-classifier configured
[102]:
predict_rest_mnist(mnist,"mnist-classifier","seldon",ISTIO_GATEWAY)
../_images/examples_istio_canary_41_0.png
<Response [200]>
Route:{}
{
  "class:0": "0.00",
  "class:1": "0.00",
  "class:2": "0.01",
  "class:3": "0.11",
  "class:4": "0.00",
  "class:5": "0.04",
  "class:6": "0.00",
  "class:7": "0.00",
  "class:8": "0.84",
  "class:9": "0.00"
}

You should now see all traffic transfer to the canary.

sk-preditor2

tf-preditor2

[ ]: