This page was generated from examples/istio/canary_update/canary.ipynb.

Canary Deployment with Seldon and Istio

Running this example

*This demo needs egress when running the load test to allow MNIST digits to be downloaded. If you want to run the load test then you will need to follow the docs on egress `here <https://istio.io/docs/tasks/traffic-management/egress/#calling-external-services-directly>`__ if you run istio in a way that egress is blocked*

Setup Cluster and Ingress

Use the setup notebook to Setup Cluster with Istio Ingress. Instructions also online.

Setup Istio

Ensure you have istio installed. Follow their docs

For this example we will create the default istio gateway for seldon which needs to be called seldon-gateway. You can supply your own gateway by adding to your SeldonDeployments resources the annotation seldon.io/istio-gateway with values the name of your istio gateway.

Create a gateway for our istio-ingress

[19]:
!kubectl create -f ../../../notebooks/resources/seldon-gateway.yaml
gateway.networking.istio.io/seldon-gateway created

Label our namespace so istio creates sidecars

[20]:
!kubectl label namespace seldon istio-injection=enabled
namespace/seldon labeled

If you are using Minikube for your Kubernetes cluster you will need to run as root in a separte terminal:

minikube tunnel

This will allow a LoadBalancer to be simulated on your local machine.

[21]:
INGRESS_HOST=!kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
INGRESS_PORT=!kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].port}'
ISTIO_GATEWAY=INGRESS_HOST[0]+":"+INGRESS_PORT[0]

Check the istio gateway address

[22]:
ISTIO_GATEWAY
[22]:
'10.109.199.85:80'

To view the istio traffic you can go to the istio grafana dashboard. In a separate terminal port-forward to it:

See their docs here

Start Seldon Core

Uset the setup notebook to Install Seldon Core with Istio Ingress. Instructions also online.

Serve Single Model

[25]:
from random import randint,random
import json
from matplotlib import pyplot as plt
import numpy as np
from tensorflow.examples.tutorials.mnist import input_data
from seldon_core.seldon_client import SeldonClient
import seldon_core

def gen_image(arr):
    two_d = (np.reshape(arr, (28, 28)) * 255).astype(np.uint8)
    plt.imshow(two_d,cmap=plt.cm.gray_r, interpolation='nearest')
    return plt

def download_mnist():
    return input_data.read_data_sets("MNIST_data/", one_hot = True)

def predict_rest_mnist(mnist,deployment_name,namespace,istio_gateway):
    sc = SeldonClient(deployment_name=deployment_name,namespace=namespace,gateway_endpoint=istio_gateway)
    batch_xs, batch_ys = mnist.train.next_batch(1)
    chosen=0
    gen_image(batch_xs[chosen]).show()
    data = batch_xs[chosen].reshape((1,784))
    features = ["X"+str(i+1) for i in range (0,784)]
    r = sc.predict(gateway="istio",transport="rest",shape=(1,784),data=data,payload_type='ndarray',names=features)
    predictions = seldon_core.utils.seldon_message_to_json(r.response)
    print(predictions)
    #print("Route:"+json.dumps(predictions["meta"]["routing"],indent=2))
    fpreds = [ '%.2f' % elem for elem in predictions["data"]["ndarray"][0] ]
    m = dict(zip(predictions["data"]["names"],fpreds))
    print(json.dumps(m,indent=2))

[26]:
%matplotlib inline
from visualizer import get_graph
mnist = download_mnist()
Extracting MNIST_data/train-images-idx3-ubyte.gz
Extracting MNIST_data/train-labels-idx1-ubyte.gz
Extracting MNIST_data/t10k-images-idx3-ubyte.gz
Extracting MNIST_data/t10k-labels-idx1-ubyte.gz
[27]:
!kubectl apply -f mnist_v1.json
seldondeployment.machinelearning.seldon.io/mnist-classifier created
[28]:
!kubectl rollout status deploy/mnist-deployment-sk-mnist-predictor-73d7608
deployment "mnist-deployment-sk-mnist-predictor-73d7608" successfully rolled out
[29]:
predict_rest_mnist(mnist,"mnist-classifier","seldon",ISTIO_GATEWAY)
../_images/examples_istio_canary_19_0.png
{'meta': {'puid': 'nn5g30lmmgj5sqg1etdcbi4695', 'requestPath': {'sk-mnist-classifier': 'seldonio/sk-example-mnist:0.2'}}, 'data': {'names': ['class:0', 'class:1', 'class:2', 'class:3', 'class:4', 'class:5', 'class:6', 'class:7', 'class:8', 'class:9'], 'ndarray': [[0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]]}}
{
  "class:0": "0.00",
  "class:1": "1.00",
  "class:2": "0.00",
  "class:3": "0.00",
  "class:4": "0.00",
  "class:5": "0.00",
  "class:6": "0.00",
  "class:7": "0.00",
  "class:8": "0.00",
  "class:9": "0.00"
}

Start a Load Test

[34]:
!kubectl label nodes $(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') role=locust
node/minikube labeled
[44]:
!helm install ../../../helm-charts/seldon-core-loadtesting --name loadtest  \
    --namespace seldon \
    --repo https://storage.googleapis.com/seldon-charts \
    --set locust.script=mnist_rest_locust.py \
    --set locust.host=http://{ISTIO_GATEWAY} \
    --set rest.pathPrefix=/seldon/seldon/mnist-classifier \
    --set oauth.enabled=false \
    --set oauth.key=oauth-key \
    --set oauth.secret=oauth-secret \
    --set locust.hatchRate=1 \
    --set locust.clients=1 \
    --set loadtest.sendFeedback=1 \
    --set locust.minWait=0 \
    --set locust.maxWait=0 \
    --set replicaCount=1 \
    --set data.size=784

NAME:   loadtest
LAST DEPLOYED: Sun Jun 30 16:23:57 2019
NAMESPACE: seldon
STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME                   READY  STATUS       RESTARTS  AGE
locust-master-1-mblc6  0/2    Init:0/1     0         1s
locust-master-1-trgc5  0/2    Terminating  0         85s
locust-slave-1-jgqp2   0/2    Init:0/1     0         1s
locust-slave-1-mdgxc   0/2    Terminating  0         85s

==> v1/ReplicationController
NAME             DESIRED  CURRENT  READY  AGE
locust-master-1  1        1        0      1s
locust-slave-1   1        1        0      1s

==> v1/Service
NAME             TYPE      CLUSTER-IP     EXTERNAL-IP  PORT(S)                                       AGE
locust-master-1  NodePort  10.98.215.152  <none>       5557:31305/TCP,5558:31110/TCP,8089:31793/TCP  1s


Now we will add a canary and split traffic 75% to 25% to it. This is done by adding a new predictor to the SeldonDeployment and specifying the traffic values.

[40]:
!pygmentize mnist_v2.json
{
    "apiVersion": "machinelearning.seldon.io/v1alpha2",
    "kind": "SeldonDeployment",
    "metadata": {
        "labels": {
            "app": "seldon"
        },
        "name": "mnist-classifier"
    },
    "spec": {
        "annotations": {
            "project_name": "Mnist classification"
        },
        "name": "mnist-deployment",
        "predictors": [
            {
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/sk-example-mnist:0.2",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "sk-mnist-classifier",
                                "resources": {
                                    "requests": {
                                        "memory": "1Mi"
                                    }
                                }
                            }
                        ],
                        "terminationGracePeriodSeconds": 1
                    }
                }],
                "graph": {
                    "children": [],
                    "name": "sk-mnist-classifier",
                    "endpoint": {
                        "type" : "REST"
                    },
                    "type": "MODEL"
                },
                "name": "sk-mnist-predictor",
                "replicas": 1,
                "labels":{
                    "version":"v1"
                },
                "traffic": 75
            },
            {
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/tf-example-mnist:0.1",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "tf-mnist-classifier",
                                "resources": {
                                    "requests": {
                                        "memory": "1Mi"
                                    }
                                }
                            }
                        ],
                        "terminationGracePeriodSeconds": 1
                    }
                }],
                "graph": {
                    "children": [],
                    "name": "tf-mnist-classifier",
                    "endpoint": {
                        "type" : "REST"
                    },
                    "type": "MODEL"
                },
                "name": "tf-mnist-predictor",
                "replicas": 1,
                "labels":{
                    "version":"v2"
                },
                "traffic": 25
            }

        ]
    }
}
[41]:
get_graph("mnist_v2.json")
[41]:
../_images/examples_istio_canary_25_0.svg
[42]:
!kubectl apply -f mnist_v2.json
seldondeployment.machinelearning.seldon.io/mnist-classifier configured
[45]:
predict_rest_mnist(mnist,"mnist-classifier","seldon",ISTIO_GATEWAY)
../_images/examples_istio_canary_27_0.png
{'meta': {'puid': 'pii2cg6n9gpke7gag6etkuj4cv', 'requestPath': {'sk-mnist-classifier': 'seldonio/sk-example-mnist:0.2'}}, 'data': {'names': ['class:0', 'class:1', 'class:2', 'class:3', 'class:4', 'class:5', 'class:6', 'class:7', 'class:8', 'class:9'], 'ndarray': [[0.0, 0.0, 0.03333333333333333, 0.8666666666666667, 0.0, 0.06666666666666667, 0.0, 0.0, 0.03333333333333333, 0.0]]}}
{
  "class:0": "0.00",
  "class:1": "0.00",
  "class:2": "0.03",
  "class:3": "0.87",
  "class:4": "0.00",
  "class:5": "0.07",
  "class:6": "0.00",
  "class:7": "0.00",
  "class:8": "0.03",
  "class:9": "0.00"
}

You should see traffic being split on the Istio service dashboard for the mnist-classifier.

skpredictor

tfpredictor

When you are happy the canary is ok you can promote to full traffic.

[46]:
!kubectl apply -f mnist_v3.json
seldondeployment.machinelearning.seldon.io/mnist-classifier configured
[102]:
predict_rest_mnist(mnist,"mnist-classifier","seldon",ISTIO_GATEWAY)
../_images/examples_istio_canary_32_0.png
<Response [200]>
Route:{}
{
  "class:0": "0.00",
  "class:1": "0.00",
  "class:2": "0.01",
  "class:3": "0.11",
  "class:4": "0.00",
  "class:5": "0.04",
  "class:6": "0.00",
  "class:7": "0.00",
  "class:8": "0.84",
  "class:9": "0.00"
}

You should now see all traffic transfer to the canary.

sk-preditor2

tf-preditor2

[ ]: