This page was generated from examples/istio/canary_update/canary.ipynb.

Canary Deployment with Seldon and Istio

Running this example

*This demo needs egress when running the load test to allow MNIST digits to be downloaded. If you want to run the load test then you will need to follow the docs on egress `here <https://istio.io/docs/tasks/traffic-management/egress/#calling-external-services-directly>`__ if you run istio in a way that egress is blocked*

Setup Cluster and Ingress

Use the setup notebook to Setup Cluster with Istio Ingress. Instructions also online.

[ ]:
!kubectl create namespace seldon
[ ]:
!kubectl config set-context $(kubectl config current-context) --namespace=seldon

Setup Istio

Ensure you have istio installed. Follow their docs

For this example we will create the default istio gateway for seldon which needs to be called seldon-gateway. You can supply your own gateway by adding to your SeldonDeployments resources the annotation seldon.io/istio-gateway with values the name of your istio gateway.

Create a gateway for our istio-ingress

[ ]:
!kubectl apply -f ../../../notebooks/resources/seldon-gateway.yaml

Ensure the istio ingress gatewaty is port-forwarded to localhost:8004

  • Istio: kubectl port-forward $(kubectl get pods -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}') -n istio-system 8004:80
[ ]:
ISTIO_GATEWAY="localhost:8004"

To view the istio traffic you can go to the istio grafana dashboard. In a separate terminal port-forward to it:

See their docs here

Serve Single Model

[ ]:
from random import randint,random
import json
from matplotlib import pyplot as plt
import numpy as np
from tensorflow.examples.tutorials.mnist import input_data
from seldon_core.seldon_client import SeldonClient
import seldon_core
import time

def gen_image(arr):
    two_d = (np.reshape(arr, (28, 28)) * 255).astype(np.uint8)
    plt.imshow(two_d,cmap=plt.cm.gray_r, interpolation='nearest')
    return plt

def download_mnist():
    return input_data.read_data_sets("MNIST_data/", one_hot = True)

def predict_rest_mnist(mnist,deployment_name,namespace,istio_gateway):
    sc = SeldonClient(deployment_name=deployment_name,namespace=namespace,gateway_endpoint=istio_gateway)
    batch_xs, batch_ys = mnist.train.next_batch(1)
    chosen=0
    gen_image(batch_xs[chosen]).show()
    data = batch_xs[chosen].reshape((1,784))
    features = ["X"+str(i+1) for i in range (0,784)]
    r = sc.predict(gateway="istio",transport="rest",shape=(1,784),data=data,payload_type='ndarray',names=features)
    predictions = r.response
    fpreds = [ '%.2f' % elem for elem in predictions["data"]["ndarray"][0] ]
    m = dict(zip(predictions["data"]["names"],fpreds))
    print(json.dumps(m,indent=2))

[ ]:
%matplotlib inline
mnist = download_mnist()
[ ]:
!kubectl apply -f mnist_v1.json
[ ]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=mnist-classifier -o jsonpath='{.items[0].metadata.name}')
[ ]:
for i in range(60):
    state=!kubectl get sdep mnist-classifier -o jsonpath='{.status.state}'
    state=state[0]
    print(state)
    if state=="Available":
        break
    time.sleep(1)
assert(state=="Available")
[ ]:
predict_rest_mnist(mnist,"mnist-classifier","seldon",ISTIO_GATEWAY)

Now we will add a canary and split traffic 75% to 25% to it. This is done by adding a new predictor to the SeldonDeployment and specifying the traffic values.

[ ]:
!pygmentize mnist_v2.json
[ ]:
!kubectl apply -f mnist_v2.json
[ ]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=mnist-classifier -o jsonpath='{.items[0].metadata.name}')
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=mnist-classifier -o jsonpath='{.items[1].metadata.name}')
[ ]:
for i in range(60):
    state=!kubectl get sdep mnist-classifier -o jsonpath='{.status.state}'
    state=state[0]
    print(state)
    if state=="Available":
        break
    time.sleep(1)
assert(state=="Available")
[ ]:
for i in range(1,100):
    predict_rest_mnist(mnist,"mnist-classifier","seldon",ISTIO_GATEWAY)

Check the traffic is split

[ ]:
default_count=!kubectl logs $(kubectl get pod -l seldon-app=mnist-classifier-sk-mnist-predictor -o jsonpath='{.items[0].metadata.name}') seldon-container-engine  | grep "/predict" | wc -l
[ ]:
canary_count=!kubectl logs $(kubectl get pod -l seldon-app=mnist-classifier-tf-mnist-predictor -o jsonpath='{.items[0].metadata.name}') seldon-container-engine  | grep "/predict" | wc -l
[ ]:
percentage_to_canary = float(canary_count[0])/float(default_count[0])
print(percentage_to_canary)
assert(percentage_to_canary > 0.1 and percentage_to_canary < 0.5)

When you are happy the canary is ok you can promote to full traffic.

[ ]:
!kubectl apply -f mnist_v3.json
[ ]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=mnist-classifier -o jsonpath='{.items[0].metadata.name}')
[ ]:
for i in range(60):
    state=!kubectl get sdep mnist-classifier -o jsonpath='{.status.state}'
    state=state[0]
    print(state)
    if state=="Creating":
        break
    time.sleep(1)
[ ]:
for i in range(60):
    state=!kubectl get sdep mnist-classifier -o jsonpath='{.status.state}'
    state=state[0]
    print(state)
    if state=="Available":
        break
    time.sleep(1)
assert(state=="Available")
[ ]:
predict_rest_mnist(mnist,"mnist-classifier","seldon",ISTIO_GATEWAY)
[ ]:
!kubectl delete -f mnist_v3.json
[ ]: