This page was generated from examples/ambassador/custom/ambassador_custom.ipynb.

Custom Header Routing with Seldon and Ambassador¶

This notebook shows how you can deploy Seldon Deployments with custom Ambassador configuration.

Prerequistes¶

You will need

Git clone of Seldon Core running this notebook
A running Kubernetes cluster with kubectl authenticated
Helm client
Seldon Core Python Module : pip install seldon-core

Creating a Kubernetes Cluster¶

Follow the Kubernetes documentation to create a cluster.

Once created ensure kubectl is authenticated against the running cluster.

Setup¶

[1]:

!kubectl create namespace seldon

namespace/seldon created

[2]:

!kubectl config set-context $(kubectl config current-context) --namespace=seldon

Context "minikube" modified.

[3]:

!kubectl create clusterrolebinding kube-system-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default

clusterrolebinding.rbac.authorization.k8s.io/kube-system-cluster-admin created

Install Helm¶

[4]:

!kubectl -n kube-system create sa tiller
!kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
!helm init --service-account tiller

serviceaccount/tiller created
clusterrolebinding.rbac.authorization.k8s.io/tiller created
$HELM_HOME has been configured at /home/clive/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!

[5]:

!kubectl rollout status deploy/tiller-deploy -n kube-system

Waiting for deployment "tiller-deploy" rollout to finish: 0 of 1 updated replicas are available...
deployment "tiller-deploy" successfully rolled out

Start seldon-core¶

[2]:

!helm install ../../../helm-charts/seldon-core-operator --name seldon-core --set usageMetrics.enabled=true --namespace seldon-system

NAME:   seldon-core
LAST DEPLOYED: Tue Apr 16 09:16:25 2019
NAMESPACE: seldon-system
STATUS: DEPLOYED

RESOURCES:
==> v1/Secret
NAME                                   TYPE    DATA  AGE
seldon-operator-webhook-server-secret  Opaque  0     0s

==> v1/ClusterRole
NAME                          AGE
seldon-operator-manager-role  0s

==> v1/ClusterRoleBinding
NAME                                 AGE
seldon-operator-manager-rolebinding  0s

==> v1beta1/Deployment
NAME                        DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
seldon-spartakus-volunteer  1        0        0           0          0s

==> v1/ServiceAccount
NAME                        SECRETS  AGE
seldon-spartakus-volunteer  1        0s

==> v1beta1/ClusterRoleBinding
NAME                        AGE
seldon-spartakus-volunteer  0s

==> v1/Pod(related)
NAME                                  READY  STATUS             RESTARTS  AGE
seldon-operator-controller-manager-0  0/1    ContainerCreating  0         0s

==> v1/ConfigMap
NAME                     DATA  AGE
seldon-spartakus-config  3     0s

==> v1beta1/CustomResourceDefinition
NAME                                         AGE
seldondeployments.machinelearning.seldon.io  0s

==> v1/Service
NAME                                        TYPE       CLUSTER-IP   EXTERNAL-IP  PORT(S)  AGE
seldon-operator-controller-manager-service  ClusterIP  10.106.7.42  <none>       443/TCP  0s

==> v1/StatefulSet
NAME                                DESIRED  CURRENT  AGE
seldon-operator-controller-manager  1        1        0s

==> v1beta1/ClusterRole
NAME                        AGE
seldon-spartakus-volunteer  0s


NOTES:
NOTES: TODO

[3]:

!kubectl rollout status statefulset.apps/seldon-operator-controller-manager -n seldon-system

partitioned roll out complete: 1 new pods have been updated...

Setup Ingress¶

There are gRPC issues with the latest Ambassador, so we rewcommend 0.40.2 until these are fixed.

[4]:

!helm install stable/ambassador --name ambassador --set image.tag=0.40.2

NAME:   ambassador
LAST DEPLOYED: Tue Apr 16 09:16:36 2019
NAMESPACE: seldon
STATUS: DEPLOYED

RESOURCES:
==> v1/ServiceAccount
NAME        SECRETS  AGE
ambassador  1        0s

==> v1beta1/ClusterRole
NAME        AGE
ambassador  0s

==> v1beta1/ClusterRoleBinding
NAME        AGE
ambassador  0s

==> v1/Service
NAME               TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)                     AGE
ambassador-admins  ClusterIP     10.110.148.36  <none>       8877/TCP                    0s
ambassador         LoadBalancer  10.102.23.33   <pending>    80:32682/TCP,443:30170/TCP  0s

==> v1/Deployment
NAME        DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
ambassador  3        3        3           0          0s

==> v1/Pod(related)
NAME                         READY  STATUS             RESTARTS  AGE
ambassador-5b89d44544-6g9fv  0/1    ContainerCreating  0         0s
ambassador-5b89d44544-8n7rp  0/1    ContainerCreating  0         0s
ambassador-5b89d44544-dwcxd  0/1    ContainerCreating  0         0s


NOTES:
Congratuations! You've successfully installed Ambassador.

For help, visit our Slack at https://d6e.co/slack or view the documentation online at https://www.getambassador.io.

To get the IP address of Ambassador, run the following commands:
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
     You can watch the status of by running 'kubectl get svc -w  --namespace seldon ambassador'

  On GKE/Azure:
  export SERVICE_IP=$(kubectl get svc --namespace seldon ambassador -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

  On AWS:
  export SERVICE_IP=$(kubectl get svc --namespace seldon ambassador -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

  echo http://$SERVICE_IP:

[5]:

!kubectl rollout status deployment.apps/ambassador

Waiting for deployment "ambassador" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "ambassador" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "ambassador" rollout to finish: 2 of 3 updated replicas are available...
deployment "ambassador" successfully rolled out

Port Forward to Ambassador¶

kubectl port-forward $(kubectl get pods -n seldon -l app.kubernetes.io/name=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8003:8080

Launch main model¶

We will create a very simple Seldon Deployment with a dummy model image seldonio/mock_classifier:1.0. This deployment is named example. We will add custom Ambassador config which sets the Ambassador prefix to /mycompany/ml

We must ensure we set the correct service endpoint. Seldon Core creates an endpoint of the form:

<spec.name>-<metadata.name>.<namespace>:<port>

Where

<spec-name> is the name you give to the Seldon Deployment spec: production-model below
<metadata.name> is the metadata name in the Seldon Deployment: example below
<namespace> is the namespace your Seldon Deployment is deployed to
<port> is the port either 8000 for REST or 5000 for gRPC

This will allow you to set the service value in the Ambassador config you create. So for the example below we have:

service: production-model-example.seldon:8000

[6]:

!pygmentize model_custom_ambassador.json

{
    "apiVersion": "machinelearning.seldon.io/v1alpha2",
    "kind": "SeldonDeployment",
    "metadata": {
        "labels": {
            "app": "seldon"
        },
        "name": "example"
    },
    "spec": {
        "name": "production-model",
        "annotations": {
            "seldon.io/ambassador-config":"apiVersion: ambassador/v0\nkind: Mapping\nname: seldon_example_rest_mapping\nprefix: /mycompany/ml/\nservice: production-model-example.seldon:8000\ntimeout_ms: 3000"
        },
        "predictors": [
            {
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/mock_classifier:1.0",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier"
                            }
                        ],
                        "terminationGracePeriodSeconds": 1
                    }}
                                  ],
                "graph":
                {
                    "children": [],
                    "name": "classifier",
                    "type": "MODEL",
                    "endpoint": {
                        "type": "REST"
                    }},
                "name": "single",
                "replicas": 1
            }
        ]
    }
}

[7]:

!kubectl create -f model_custom_ambassador.json

seldondeployment.machinelearning.seldon.io/example created

[8]:

!kubectl rollout status deploy/production-model-single-7cd068f

Waiting for deployment "production-model-single-7cd068f" rollout to finish: 0 of 1 updated replicas are available...
deployment "production-model-single-7cd068f" successfully rolled out

Get predictions¶

[9]:

from seldon_core.seldon_client import SeldonClient
sc = SeldonClient(deployment_name="example",namespace="seldon")

REST Request¶

[10]:

r = sc.predict(gateway="ambassador",transport="rest",ambassador_prefix="/mycompany/ml")
print(r)

Success:True message:
Request:
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.1628281752658065
  }
}

Response:
meta {
  puid: "5lb8edtgn4rhbd03g94g318io7"
  requestPath {
    key: "classifier"
    value: "seldonio/mock_classifier:1.0"
  }
}
data {
  names: "proba"
  tensor {
    shape: 1
    shape: 1
    values: 0.059870232304306846
  }
}

[ ]: