Custom Header Routing with Seldon and Ambassador

This notebook shows how you can deploy Seldon Deployments which can have custom routing via Ambassador’s custom header routing.

Setup Seldon Core

Use the setup notebook to Setup Cluster with Ambassador Ingress and Install Seldon Core. Instructions also online.

[ ]:
!kubectl create namespace seldon
[ ]:
!kubectl config set-context $(kubectl config current-context) --namespace=seldon

Launch main model

We will create a very simple Seldon Deployment with a dummy model image seldonio/mock_classifier:1.0. This deployment is named example.

[ ]:
!pygmentize model.json
[ ]:
!kubectl create -f model.json
[ ]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=example -o jsonpath='{.items[0].metadata.name}')

Get predictions

[ ]:
from seldon_core.seldon_client import SeldonClient
sc = SeldonClient(deployment_name="example",namespace="seldon")

REST Request

[ ]:
r = sc.predict(gateway="ambassador",transport="rest")
assert(r.success==True)
print(r)

Launch Model with Custom Routing

We will now create a new graph for our Canary with a new model seldonio/mock_classifier_rest:1.1. To make it a canary of the original example deployment we add two annotations

"annotations": {
        "seldon.io/ambassador-header":"location:london"
        "seldon.io/ambassador-service-name":"example"
    },

The first annotation says we want to route traffic that has the header location:london. The second says we want to use example as our service endpoint rather than the default which would be our deployment name - in this case example-canary. This will ensure that this Ambassador setting will apply to the same prefix as the previous one.

[ ]:
!pygmentize model_with_header.json
[ ]:
!kubectl create -f model_with_header.json
[ ]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=example-header -o jsonpath='{.items[0].metadata.name}')

Check a request without a header goes to the existing model.

[ ]:
r = sc.predict(gateway="ambassador",transport="rest")
print(r)
[ ]:
default_count=!kubectl logs $(kubectl get pod -lseldon-app=example-single -o jsonpath='{.items[0].metadata.name}') classifier | grep "/predict" | wc -l
[ ]:
print(default_count)
assert(int(default_count[0]) == 2)

Check a REST request with the required header gets routed to the new model.

[ ]:
r = sc.predict(gateway="ambassador",transport="rest",headers={"location":"london"})
print(r)
[ ]:
header_count=!kubectl logs $(kubectl get pod -lseldon-app=example-header-single -o jsonpath='{.items[0].metadata.name}') classifier | grep "/predict" | wc -l
[ ]:
print(header_count)
assert(int(header_count[0]) == 1)
[ ]:
!kubectl delete -f model.json
[ ]:
!kubectl delete -f model_with_header.json
[ ]: