This page was generated from notebooks/istio_example.ipynb.

Example Seldon Core Deployments using Helm

Prerequistes

You will need - Git clone of Seldon Core - A running Kubernetes cluster with kubectl authenticated - seldon-core Python package (pip install seldon-core>=0.2.6.1) - Helm client

Creating a Kubernetes Cluster

Follow the Kubernetes documentation to create a cluster.

Once created ensure kubectl is authenticated against the running cluster.

Setup

[22]:
!kubectl create namespace seldon
namespace/seldon created
[23]:
!kubectl config set-context $(kubectl config current-context) --namespace=seldon
Context "minikube" modified.
[24]:
!kubectl create clusterrolebinding kube-system-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
clusterrolebinding.rbac.authorization.k8s.io/kube-system-cluster-admin created

Install Helm

[25]:
!kubectl -n kube-system create sa tiller
!kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
!helm init --service-account tiller
serviceaccount/tiller created
clusterrolebinding.rbac.authorization.k8s.io/tiller created
$HELM_HOME has been configured at /home/clive/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!
[26]:
!kubectl rollout status deploy/tiller-deploy -n kube-system
Waiting for deployment "tiller-deploy" rollout to finish: 0 of 1 updated replicas are available...
deployment "tiller-deploy" successfully rolled out

Setup Istio

Ensure you have istio installed. Follow their docs

For this example we will create the default istio gateway for seldon which needs to be called seldon-gateway. You can supply your own gateway by adding to your SeldonDeployments resources the annotation seldon.io/istio-gateway with values the name of your istio gateway.

Create a gateway for our istio-ingress

[27]:
!kubectl create -f resources/seldon-gateway.yaml
gateway.networking.istio.io/seldon-gateway created

Label our namespace so istio creates sidecars

[28]:
!kubectl label namespace seldon istio-injection=enabled
namespace/seldon labeled

If you are using Minikube for your Kubernetes cluster you will need to run as root in a separte terminal:

minikube tunnel

This will allow a LoadBalancer to be simulated on your local machine.

[29]:
INGRESS_HOST=!kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
INGRESS_PORT=!kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].port}'
ISTIO_GATEWAY=INGRESS_HOST[0]+":"+INGRESS_PORT[0]
[30]:
ISTIO_GATEWAY
[30]:
'10.97.234.81:80'

Start seldon-core

[31]:
!helm install ../helm-charts/seldon-core-operator --name seldon-core --set istio.enabled=true --set usageMetrics.enabled=true --namespace seldon-system
NAME:   seldon-core
LAST DEPLOYED: Sat May 25 11:33:25 2019
NAMESPACE: seldon-system
STATUS: DEPLOYED

RESOURCES:
==> v1/ClusterRole
NAME                          AGE
seldon-operator-manager-role  1s

==> v1/ClusterRoleBinding
NAME                                 AGE
seldon-operator-manager-rolebinding  1s

==> v1/ConfigMap
NAME                     DATA  AGE
seldon-spartakus-config  3     1s

==> v1/Pod(related)
NAME                                         READY  STATUS             RESTARTS  AGE
seldon-operator-controller-manager-0         0/1    ContainerCreating  0         1s
seldon-spartakus-volunteer-5554c4d8b6-hvmt2  0/1    ContainerCreating  0         0s

==> v1/Secret
NAME                                   TYPE    DATA  AGE
seldon-operator-webhook-server-secret  Opaque  0     1s

==> v1/Service
NAME                                        TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)  AGE
seldon-operator-controller-manager-service  ClusterIP  10.103.145.68  <none>       443/TCP  1s

==> v1/ServiceAccount
NAME                        SECRETS  AGE
seldon-spartakus-volunteer  1        1s

==> v1/StatefulSet
NAME                                READY  AGE
seldon-operator-controller-manager  0/1    1s

==> v1beta1/ClusterRole
NAME                        AGE
seldon-spartakus-volunteer  1s

==> v1beta1/ClusterRoleBinding
NAME                        AGE
seldon-spartakus-volunteer  0s

==> v1beta1/CustomResourceDefinition
NAME                                         AGE
seldondeployments.machinelearning.seldon.io  1s

==> v1beta1/Deployment
NAME                        READY  UP-TO-DATE  AVAILABLE  AGE
seldon-spartakus-volunteer  0/1    1           0          1s


NOTES:
NOTES: TODO


[32]:
!kubectl rollout status statefulset.apps/seldon-operator-controller-manager -n seldon-system
partitioned roll out complete: 1 new pods have been updated...

Serve Single Model

[33]:
!helm install ../helm-charts/seldon-single-model --name mymodel
NAME:   mymodel
LAST DEPLOYED: Sat May 25 11:36:23 2019
NAMESPACE: seldon
STATUS: DEPLOYED

RESOURCES:
==> v1alpha2/SeldonDeployment
NAME     AGE
mymodel  0s


[34]:
!helm template ../helm-charts/seldon-single-model | pygmentize -l json
---
# Source: seldon-single-model/templates/model.json
{
    "apiVersion": "machinelearning.seldon.io/v1alpha2",
    "kind": "SeldonDeployment",
    "metadata": {
        "labels": {
            "app": "seldon"
        },
        "name": "release-name"
    },
    "spec": {
        "name": "release-name",
        "predictors": [
            {
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/mock_classifier:1.0",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier",
                                "resources": {
                                    "requests": {
                                        "memory": "1Mi"
                                    }
                                }
                            }
                        ],
                        "terminationGracePeriodSeconds": 1
                    }}
                                  ],
                "graph":
                {
                    "children": [],
                    "name": "classifier",
                    "type": "MODEL",
                    "endpoint": {
                        "type": "REST"
                    }},
                "name": "release-name",
                "replicas": 1,
            "labels": {
            "version" : "v1"
            }
            }
        ]
    }
}
[35]:
!kubectl rollout status deploy/mymodel-mymodel-7cd068f
deployment "mymodel-mymodel-7cd068f" successfully rolled out

Get predictions

[36]:
from seldon_core.seldon_client import SeldonClient
sc = SeldonClient(deployment_name="mymodel",namespace="seldon",gateway_endpoint=ISTIO_GATEWAY)

REST Request

[37]:
r = sc.predict(gateway="istio",transport="rest")
print(r)
Success:True message:
Request:
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.17442955533025983
  }
}

Response:
meta {
  puid: "q51i2q9ri9scl4uqi9bjrkr25f"
  requestPath {
    key: "classifier"
    value: "seldonio/mock_classifier:1.0"
  }
}
data {
  names: "proba"
  tensor {
    shape: 1
    shape: 1
    values: 0.060526569086473254
  }
}

gRPC Request

[38]:
r = sc.predict(gateway="istio",transport="grpc")
print(r)
Success:True message:
Request:
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.876372699790597
  }
}

Response:
meta {
  puid: "hfd50d101q22qkjpme5t1r3b92"
  requestPath {
    key: "classifier"
    value: "seldonio/mock_classifier:1.0"
  }
}
data {
  names: "proba"
  tensor {
    shape: 1
    shape: 1
    values: 0.11503680184999401
  }
}

[39]:
!helm delete mymodel --purge
release "mymodel" deleted

Serve AB Test

[8]:
!helm install ../helm-charts/seldon-abtest --name myabtest
NAME:   myabtest
LAST DEPLOYED: Sat May 25 11:11:01 2019
NAMESPACE: seldon
STATUS: DEPLOYED

RESOURCES:
==> v1alpha2/SeldonDeployment
NAME      AGE
myabtest  0s


[9]:
!helm template ../helm-charts/seldon-abtest | pygmentize -l json
---
# Source: seldon-abtest/templates/ab_test_1pod.json


---
# Source: seldon-abtest/templates/ab_test_2pods.json

{
    "apiVersion": "machinelearning.seldon.io/v1alpha2",
    "kind": "SeldonDeployment",
    "metadata": {
        "labels": {
            "app": "seldon"
        },
        "name": "release-name"
    },
    "spec": {
        "name": "release-name",
        "predictors": [
            {
                "name": "abtest",
                "replicas": 1,
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/mock_classifier:1.0",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier-1",
                                "resources": {
                                    "requests": {
                                        "memory": "1Mi"
                                    }
                                }
                            }],
                        "terminationGracePeriodSeconds": 20
                    }},
                {
                    "metadata":{
                        "labels":{
                            "version":"v2"
                        }
                    },
                        "spec":{
                            "containers":[
                            {
                                "image": "seldonio/mock_classifier:1.0",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier-2",
                                "resources": {
                                    "requests": {
                                        "memory": "1Mi"
                                    }
                                }
                            }
                        ],
                        "terminationGracePeriodSeconds": 20
                                   }
                                   }],
                "graph": {
                    "name": "release-name",
                    "endpoint":{},
                    "implementation":"RANDOM_ABTEST",
                    "parameters": [
                        {
                            "name":"ratioA",
                            "value":"0.5",
                            "type":"FLOAT"
                        }
                    ],
                    "children": [
                        {
                            "name": "classifier-1",
                            "endpoint":{
                                "type":"REST"
                            },
                            "type":"MODEL",
                            "children":[]
                        },
                        {
                            "name": "classifier-2",
                            "endpoint":{
                                "type":"REST"
                            },
                            "type":"MODEL",
                            "children":[]
                        }
                    ]
                }
            }
        ]
    }
}



[10]:
!kubectl rollout status deploy/myabtest-abtest-41de5b8
!kubectl rollout status deploy/myabtest-abtest-df66c5c
Waiting for deployment "myabtest-abtest-41de5b8" rollout to finish: 0 of 1 updated replicas are available...
deployment "myabtest-abtest-41de5b8" successfully rolled out
deployment "myabtest-abtest-df66c5c" successfully rolled out

Get predictions

[11]:
from seldon_core.seldon_client import SeldonClient
sc = SeldonClient(deployment_name="myabtest",namespace="seldon",gateway_endpoint=ISTIO_GATEWAY)

REST Request

[12]:
r = sc.predict(gateway="istio",transport="rest")
print(r)
Success:True message:
Request:
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.5812964139653322
  }
}

Response:
meta {
  puid: "qik8lbat9jfhl6e1uf9u3nlm2s"
  routing {
    key: "myabtest"
    value: 1
  }
  requestPath {
    key: "classifier-2"
    value: "seldonio/mock_classifier:1.0"
  }
  requestPath {
    key: "myabtest"
    value: ""
  }
}
data {
  names: "proba"
  tensor {
    shape: 1
    shape: 1
    values: 0.08823566926521886
  }
}

gRPC Request

[13]:
r = sc.predict(gateway="istio",transport="grpc")
print(r)
Success:True message:
Request:
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.7479179869186225
  }
}

Response:
meta {
  puid: "n2q4a586hp3aen91a0k26kiskd"
  routing {
    key: "myabtest"
    value: 0
  }
  requestPath {
    key: "classifier-1"
    value: "seldonio/mock_classifier:1.0"
  }
  requestPath {
    key: "myabtest"
    value: ""
  }
}
data {
  names: "proba"
  tensor {
    shape: 1
    shape: 1
    values: 0.10259218150165016
  }
}

[14]:
!helm delete myabtest --purge
release "myabtest" deleted

Serve Multi-Armed Bandit

[15]:
!helm install ../helm-charts/seldon-mab --name mymab
NAME:   mymab
LAST DEPLOYED: Sat May 25 11:12:08 2019
NAMESPACE: seldon
STATUS: DEPLOYED

RESOURCES:
==> v1alpha2/SeldonDeployment
NAME   AGE
mymab  1s


[16]:
!helm template ../helm-charts/seldon-mab | pygmentize -l json
---
# Source: seldon-mab/templates/mab.json
{
    "apiVersion": "machinelearning.seldon.io/v1alpha2",
    "kind": "SeldonDeployment",
    "metadata": {
        "labels": {
            "app": "seldon"
        },
        "name": "release-name"
    },
    "spec": {
        "name": "release-name",
        "predictors": [
            {
                "name": "abtest",
                "replicas": 1,
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/mock_classifier:1.0",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier-1",
                                "resources": {
                                    "requests": {
                                        "memory": "1Mi"
                                    }
                                }
                            }],
                        "terminationGracePeriodSeconds": 20
                    }},
                {
                    "metadata":{
                        "labels":{
                            "version":"v2"
                        }
                    },
                        "spec":{
                            "containers":[
                            {
                                "image": "seldonio/mock_classifier:1.0",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier-2",
                                "resources": {
                                    "requests": {
                                        "memory": "1Mi"
                                    }
                                }
                            }
                        ],
                        "terminationGracePeriodSeconds": 20
                        }
                },
                {
                    "spec":{
                        "containers": [{
                            "image": "seldonio/mab_epsilon_greedy:1.1",
                            "name": "eg-router"
                        }],
                        "terminationGracePeriodSeconds": 20
                    }}
                ],
                "graph": {
                    "name": "eg-router",
                    "type":"ROUTER",
                    "parameters": [
                        {
                            "name": "n_branches",
                            "value": "2",
                            "type": "INT"
                        },
                        {
                            "name": "epsilon",
                            "value": "0.2",
                            "type": "FLOAT"
                        },
                        {
                            "name": "verbose",
                            "value": "1",
                            "type": "BOOL"
                        }
                    ],
                    "children": [
                        {
                            "name": "classifier-1",
                            "endpoint":{
                                "type":"REST"
                            },
                            "type":"MODEL",
                            "children":[]
                        },
                        {
                            "name": "classifier-2",
                            "endpoint":{
                                "type":"REST"
                            },
                            "type":"MODEL",
                            "children":[]
                        }
                    ]
                }
            }
        ]
    }
}



[17]:
!kubectl rollout status deploy/mymab-abtest-41de5b8
!kubectl rollout status deploy/mymab-abtest-b8038b2
!kubectl rollout status deploy/mymab-abtest-df66c5c
Waiting for deployment "mymab-abtest-41de5b8" rollout to finish: 0 of 1 updated replicas are available...
deployment "mymab-abtest-41de5b8" successfully rolled out
Waiting for deployment "mymab-abtest-b8038b2" rollout to finish: 0 of 1 updated replicas are available...
deployment "mymab-abtest-b8038b2" successfully rolled out
deployment "mymab-abtest-df66c5c" successfully rolled out

Get predictions

[18]:
from seldon_core.seldon_client import SeldonClient
sc = SeldonClient(deployment_name="mymab",namespace="seldon",gateway_endpoint=ISTIO_GATEWAY)

REST Request

[19]:
r = sc.predict(gateway="istio",transport="rest")
print(r)
Success:True message:
Request:
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.48031921945868383
  }
}

Response:
meta {
  puid: "ppp3u5i1q5gl27oo16pag7h257"
  routing {
    key: "eg-router"
    value: 1
  }
  requestPath {
    key: "classifier-2"
    value: "seldonio/mock_classifier:1.0"
  }
  requestPath {
    key: "eg-router"
    value: "seldonio/mab_epsilon_greedy:1.1"
  }
}
data {
  names: "proba"
  tensor {
    shape: 1
    shape: 1
    values: 0.08044268384752119
  }
}

gRPC Request

[20]:
r = sc.predict(gateway="istio",transport="grpc")
print(r)
Success:True message:
Request:
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.16200320120248235
  }
}

Response:
meta {
  puid: "7a936sho0ajsq17q2q62f06j0p"
  routing {
    key: "eg-router"
    value: 0
  }
  requestPath {
    key: "classifier-1"
    value: "seldonio/mock_classifier:1.0"
  }
  requestPath {
    key: "eg-router"
    value: "seldonio/mab_epsilon_greedy:1.1"
  }
}
data {
  names: "proba"
  tensor {
    shape: 1
    shape: 1
    values: 0.05982381484602229
  }
}

[21]:
!helm delete mymab --purge
release "mymab" deleted
[ ]: