Canary Rollout with Seldon and Ambassador

Setup Seldon Core

Use the setup notebook to Setup Cluster with Ambassador Ingress and Install Seldon Core. Instructions also online.

[1]:
!kubectl create namespace seldon
Error from server (AlreadyExists): namespaces "seldon" already exists
[2]:
!kubectl config set-context $(kubectl config current-context) --namespace=seldon
Context "kind-seldon" modified.
[3]:
from IPython.core.magic import register_line_cell_magic

@register_line_cell_magic
def writetemplate(line, cell):
    with open(line, 'w') as f:
        f.write(cell.format(**globals()))
[4]:
VERSION=!cat ../../../version.txt
VERSION=VERSION[0]
VERSION
[4]:
'1.6.0-dev'

Launch main model

We will create a very simple Seldon Deployment with a dummy model image seldonio/mock_classifier:1.0. This deployment is named example.

[5]:
%%writetemplate model.yaml
apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  labels:
    app: seldon
  name: example
spec:
  name: canary-example
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: seldonio/mock_classifier:{VERSION}
          imagePullPolicy: IfNotPresent
          name: classifier
        terminationGracePeriodSeconds: 1
    graph:
      children: []
      endpoint:
        type: REST
      name: classifier
      type: MODEL
    name: main
    replicas: 1

[6]:
!kubectl create -f model.yaml
seldondeployment.machinelearning.seldon.io/example created
[7]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=example -o jsonpath='{.items[0].metadata.name}')
Waiting for deployment "example-main-0-classifier" rollout to finish: 0 of 1 updated replicas are available...
deployment "example-main-0-classifier" successfully rolled out

Get predictions

[8]:
from seldon_core.seldon_client import SeldonClient
sc = SeldonClient(deployment_name="example",namespace="seldon")

REST Request

[9]:
r = sc.predict(gateway="ambassador",transport="rest")
assert(r.success==True)
print(r)
Success:True message:
Request:
meta {
}
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.71309135369705
  }
}

Response:
{'data': {'names': ['proba'], 'tensor': {'shape': [1, 1], 'values': [0.09942988397161616]}}, 'meta': {'requestPath': {'classifier': 'seldonio/mock_classifier:1.6.0-dev'}}}

Launch Canary

We will now extend the existing graph and add a new predictor as a canary using a new model seldonio/mock_classifier_rest:1.1. We will add traffic values to split traffic 75/25 to the main and canary.

[10]:
%%writetemplate canary.yaml
apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  labels:
    app: seldon
  name: example
spec:
  name: canary-example
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: seldonio/mock_classifier:{VERSION}
          imagePullPolicy: IfNotPresent
          name: classifier
        terminationGracePeriodSeconds: 1
    graph:
      children: []
      endpoint:
        type: REST
      name: classifier
      type: MODEL
    name: main
    replicas: 1
    traffic: 75
  - componentSpecs:
    - spec:
        containers:
        - image: seldonio/mock_classifier:{VERSION}
          imagePullPolicy: IfNotPresent
          name: classifier
        terminationGracePeriodSeconds: 1
    graph:
      children: []
      endpoint:
        type: REST
      name: classifier
      type: MODEL
    name: canary
    replicas: 1
    traffic: 25

[11]:
!kubectl apply -f canary.yaml
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
seldondeployment.machinelearning.seldon.io/example configured
[12]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=example -o jsonpath='{.items[0].metadata.name}')
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=example -o jsonpath='{.items[1].metadata.name}')
Waiting for deployment "example-canary-0-classifier" rollout to finish: 0 of 1 updated replicas are available...
deployment "example-canary-0-classifier" successfully rolled out
deployment "example-main-0-classifier" successfully rolled out

Show our REST requests are now split with roughly 25% going to the canary.

[13]:
sc.predict(gateway="ambassador",transport="rest")
[13]:
Success:True message:
Request:
meta {
}
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.8694648161558428
  }
}

Response:
{'data': {'names': ['proba'], 'tensor': {'shape': [1, 1], 'values': [0.11433542420010957]}}, 'meta': {'requestPath': {'classifier': 'seldonio/mock_classifier:1.6.0-dev'}}}
[24]:
from collections import defaultdict
counts = defaultdict(int)
n = 100
for i in range(n):
    r = sc.predict(gateway="ambassador",transport="rest")

Following checks number of prediction requests processed by default/canary predictors respectively.

[25]:
default_count=!kubectl logs $(kubectl get pod -lseldon-app=example-main -o jsonpath='{.items[0].metadata.name}') classifier | grep "root:predict" | wc -l
[27]:
canary_count=!kubectl logs $(kubectl get pod -lseldon-app=example-canary -o jsonpath='{.items[0].metadata.name}') classifier | grep "root:predict" | wc -l
[29]:
canary_percentage=float(canary_count[0])/float(default_count[0])
print(canary_percentage)
assert(canary_percentage > 0.1 and canary_percentage < 0.5)
0.32894736842105265
[30]:
!kubectl delete -f canary.yaml
seldondeployment.machinelearning.seldon.io "example" deleted