Shadow Rollout with Seldon and Ambassador

This notebook shows how you can deploy “shadow” deployments to direct traffic not only to the main Seldon Deployment but also to a shadow deployment whose reponse will be dicarded. This allows you to test new models in a production setting and with production traffic and anlalyse how they perform before putting them live.

These are useful when you want to test a new model or higher latency inference piepline (e.g., with explanation components) with production traffic but without affecting the live deployment.

Setup Seldon Core

Use the setup notebook to Setup Cluster with Ambassador Ingress and Install Seldon Core. Instructions also online.

[ ]:
!kubectl create namespace seldon
[ ]:
!kubectl config set-context $(kubectl config current-context) --namespace=seldon

Launch main model

We will create a very simple Seldon Deployment with a dummy model image seldonio/mock_classifier:1.0. This deployment is named example.

[ ]:
!pygmentize model.json
[ ]:
!kubectl apply -f model.json
[ ]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=example -o jsonpath='{.items[0].metadata.name}')

Get predictions

[ ]:
from seldon_core.seldon_client import SeldonClient
sc = SeldonClient(deployment_name="example",namespace="seldon")

REST Request

[ ]:
r = sc.predict(gateway="ambassador",transport="rest")
print(r)

Launch Shadow

We will now create a new Seldon Deployment for our Shadow deployment with a new model.

[ ]:
!pygmentize shadow.json
[ ]:
!kubectl apply -f shadow.json
[ ]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=example -o jsonpath='{.items[0].metadata.name}')
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=example -o jsonpath='{.items[1].metadata.name}')

Let’s send a bunch of requests to the endpoint.

[ ]:
for i in range(10):
    r = sc.predict(gateway="ambassador",transport="rest")
[ ]:
default_count=!kubectl logs $(kubectl get pod -lseldon-app=example-default -o jsonpath='{.items[0].metadata.name}') classifier | grep "/predict" | wc -l
[ ]:
shadow_count=!kubectl logs $(kubectl get pod -lseldon-app=example-shadow -o jsonpath='{.items[0].metadata.name}') classifier | grep "/predict" | wc -l
[ ]:
print(shadow_count)
print(default_count)
assert(int(shadow_count[0])==10)
assert(int(default_count[0])==11)

Return to previous model

[ ]:
!kubectl apply -f model.json

TearDown

[ ]:
!kubectl delete -f model.json