Canary Rollout with Seldon and Ambassador

Setup Seldon Core

Use the setup notebook to Setup Cluster with Ambassador Ingress and Install Seldon Core. Instructions also online.

Launch main model

We will create a very simple Seldon Deployment with a dummy model image seldonio/mock_classifier:1.0. This deployment is named example.

[10]:
!pygmentize model.json
{
    "apiVersion": "machinelearning.seldon.io/v1alpha2",
    "kind": "SeldonDeployment",
    "metadata": {
        "labels": {
            "app": "seldon"
        },
        "name": "example"
    },
    "spec": {
        "name": "production-model",
        "predictors": [
            {
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/mock_classifier:1.0",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier"
                            }
                        ],
                        "terminationGracePeriodSeconds": 1
                    }}
                                  ],
                "graph":
                {
                    "children": [],
                    "name": "classifier",
                    "type": "MODEL",
                    "endpoint": {
                        "type": "REST"
                    }},
                "name": "main",
                "replicas": 1
            }
        ]
    }
}
[22]:
!kubectl create -f model.json
seldondeployment.machinelearning.seldon.io/example created
[24]:
!kubectl rollout status deploy/canary-example-main-7cd068f
Waiting for deployment "canary-example-main-7cd068f" rollout to finish: 0 of 1 updated replicas are available...
deployment "canary-example-main-7cd068f" successfully rolled out

Get predictions

[25]:
from seldon_core.seldon_client import SeldonClient
sc = SeldonClient(deployment_name="example",namespace="seldon")

REST Request

[26]:
r = sc.predict(gateway="ambassador",transport="rest")
print(r)
Success:True message:
Request:
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.6010146752277817
  }
}

Response:
meta {
  puid: "s4rqgg7i9cu4a1emd59m01rujd"
  requestPath {
    key: "classifier"
    value: "seldonio/mock_classifier:1.0"
  }
}
data {
  names: "proba"
  tensor {
    shape: 1
    shape: 1
    values: 0.08983493916158691
  }
}

gRPC Request

[36]:
r = sc.predict(gateway="ambassador",transport="grpc")
print(r)
Success:True message:
Request:
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.9518124169318304
  }
}

Response:
meta {
  puid: "mplvldoiter3si62gulsmahs58"
  requestPath {
    key: "classifier"
    value: "seldonio/mock_classifier_rest:1.1"
  }
}
data {
  names: "proba"
  tensor {
    shape: 1
    shape: 1
    values: 0.12294266589479223
  }
}

Launch Canary

We will now extend the existing graph and add a new predictor as a canary using a new model seldonio/mock_classifier_rest:1.1. We will add traffic values to split traffic 75/25 to the main and canary.

[29]:
!pygmentize canary.json
{
    "apiVersion": "machinelearning.seldon.io/v1alpha2",
    "kind": "SeldonDeployment",
    "metadata": {
        "labels": {
            "app": "seldon"
        },
        "name": "example"
    },
    "spec": {
        "name": "canary-example",
        "predictors": [
            {
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/mock_classifier:1.0",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier"
                            }
                        ],
                        "terminationGracePeriodSeconds": 1
                    }}
                                  ],
                "graph":
                {
                    "children": [],
                    "name": "classifier",
                    "type": "MODEL",
                    "endpoint": {
                        "type": "REST"
                    }},
                "name": "main",
                "replicas": 1,
                "traffic": 75
            },
            {
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/mock_classifier_rest:1.1",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier"
                            }
                        ],
                        "terminationGracePeriodSeconds": 1
                    }}
                                  ],
                "graph":
                {
                    "children": [],
                    "name": "classifier",
                    "type": "MODEL",
                    "endpoint": {
                        "type": "REST"
                    }},
                "name": "canary",
                "replicas": 1,
                "traffic": 25
            }
        ]
    }
}
[30]:
!kubectl apply -f canary.json
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
seldondeployment.machinelearning.seldon.io/example configured
[33]:
!kubectl rollout status deploy/canary-example-main-7cd068f
!kubectl rollout status deploy/canary-example-canary-4c8805f
deployment "canary-example-main-7cd068f" successfully rolled out
deployment "canary-example-canary-4c8805f" successfully rolled out

Show our REST requests are now split with roughly 25% going to the canary.

[42]:
from collections import defaultdict
counts = defaultdict(int)
n = 100
for i in range(n):
    r = sc.predict(gateway="ambassador",transport="rest")
    counts[r.response.meta.requestPath["classifier"]] += 1
for k in counts:
    print(k,(counts[k]/float(n))*100,"%")

seldonio/mock_classifier:1.0 81.0 %
seldonio/mock_classifier_rest:1.1 19.0 %

Now lets test gRPC

[37]:
counts = defaultdict(int)
n = 100
for i in range(n):
    r = sc.predict(gateway="ambassador",transport="grpc")
    counts[r.response.meta.requestPath["classifier"]] += 1
for k in counts:
    print(k,(counts[k]/float(n))*100,"%")

seldonio/mock_classifier:1.0 75.0 %
seldonio/mock_classifier_rest:1.1 25.0 %
[ ]: