This page was generated from notebooks/helm_examples.ipynb.

Example Seldon Core Deployments using Helm

Prerequistes

You will need - Git clone of Seldon Core - A running Kubernetes cluster with kubectl authenticated - seldon-core Python package (pip install seldon-core>=0.2.6.1) - Helm client

Creating a Kubernetes Cluster

Follow the Kubernetes documentation to create a cluster.

Once created ensure kubectl is authenticated against the running cluster.

Setup

[53]:
!kubectl create namespace seldon
namespace/seldon created
[54]:
!kubectl config set-context $(kubectl config current-context) --namespace=seldon
Context "minikube" modified.
[55]:
!kubectl create clusterrolebinding kube-system-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
clusterrolebinding.rbac.authorization.k8s.io/kube-system-cluster-admin created

Install Helm

[56]:
!kubectl -n kube-system create sa tiller
!kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
!helm init --service-account tiller
serviceaccount/tiller created
clusterrolebinding.rbac.authorization.k8s.io/tiller created
$HELM_HOME has been configured at /home/clive/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!
[57]:
!kubectl rollout status deploy/tiller-deploy -n kube-system
Waiting for deployment "tiller-deploy" rollout to finish: 0 of 1 updated replicas are available...
deployment "tiller-deploy" successfully rolled out

Start seldon-core

[7]:
!helm install ../helm-charts/seldon-core-operator --name seldon-core  --set usageMetrics.enabled=true --namespace seldon-system
NAME:   seldon-core
LAST DEPLOYED: Wed Aug 14 08:49:32 2019
NAMESPACE: seldon-system
STATUS: DEPLOYED

RESOURCES:
==> v1/ClusterRole
NAME                          AGE
seldon-operator-manager-role  0s

==> v1/ClusterRoleBinding
NAME                                 AGE
seldon-operator-manager-rolebinding  0s

==> v1/ConfigMap
NAME                     DATA  AGE
seldon-config            1     1s
seldon-spartakus-config  1     1s

==> v1/Pod(related)
NAME                                        READY  STATUS             RESTARTS  AGE
seldon-operator-controller-manager-0        0/1    Pending            0         0s
seldon-spartakus-volunteer-df68b9d48-cp5mg  0/1    ContainerCreating  0         0s

==> v1/Secret
NAME                                   TYPE    DATA  AGE
seldon-operator-webhook-server-secret  Opaque  0     1s

==> v1/Service
NAME                                        TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)  AGE
seldon-operator-controller-manager-service  ClusterIP  10.101.33.48   <none>       443/TCP  0s
webhook-server-service                      ClusterIP  10.103.74.125  <none>       443/TCP  0s

==> v1/ServiceAccount
NAME                              SECRETS  AGE
seldon-core-seldon-core-operator  1        1s
seldon-spartakus-volunteer        1        1s

==> v1/StatefulSet
NAME                                READY  AGE
seldon-operator-controller-manager  0/1    0s

==> v1beta1/ClusterRole
NAME                        AGE
seldon-spartakus-volunteer  0s

==> v1beta1/ClusterRoleBinding
NAME                        AGE
seldon-spartakus-volunteer  0s

==> v1beta1/CustomResourceDefinition
NAME                                         AGE
seldondeployments.machinelearning.seldon.io  1s

==> v1beta1/Deployment
NAME                        READY  UP-TO-DATE  AVAILABLE  AGE
seldon-spartakus-volunteer  0/1    1           0          0s


NOTES:
NOTES: TODO


[8]:
!kubectl rollout status statefulset.apps/seldon-operator-controller-manager -n seldon-system
partitioned roll out complete: 1 new pods have been updated...

Setup Ingress

Please note: There are reported gRPC issues with ambassador (see https://github.com/SeldonIO/seldon-core/issues/473).

[58]:
!helm install stable/ambassador --name ambassador --set crds.keep=false --set replicaCount=1
NAME:   ambassador
LAST DEPLOYED: Wed Aug 14 09:19:57 2019
NAMESPACE: seldon
STATUS: DEPLOYED

RESOURCES:
==> v1/Deployment
NAME        READY  UP-TO-DATE  AVAILABLE  AGE
ambassador  0/1    1           0          0s

==> v1/Pod(related)
NAME                         READY  STATUS             RESTARTS  AGE
ambassador-587c6cfc9f-mpcbh  0/1    ContainerCreating  0         0s

==> v1/Service
NAME              TYPE          CLUSTER-IP    EXTERNAL-IP  PORT(S)                     AGE
ambassador        LoadBalancer  10.96.198.14  <pending>    80:32010/TCP,443:32477/TCP  0s
ambassador-admin  ClusterIP     10.108.153.0  <none>       8877/TCP                    0s

==> v1/ServiceAccount
NAME        SECRETS  AGE
ambassador  1        0s

==> v1beta1/ClusterRole
NAME             AGE
ambassador       0s
ambassador-crds  0s

==> v1beta1/ClusterRoleBinding
NAME             AGE
ambassador       0s
ambassador-crds  0s

==> v1beta1/CustomResourceDefinition
NAME                                          AGE
authservices.getambassador.io                 0s
consulresolvers.getambassador.io              0s
kubernetesendpointresolvers.getambassador.io  0s
kubernetesserviceresolvers.getambassador.io   0s
mappings.getambassador.io                     0s
modules.getambassador.io                      0s
ratelimitservices.getambassador.io            0s
tcpmappings.getambassador.io                  0s
tlscontexts.getambassador.io                  0s
tracingservices.getambassador.io              0s


NOTES:
Congratuations! You've successfully installed Ambassador.

For help, visit our Slack at https://d6e.co/slack or view the documentation online at https://www.getambassador.io.

To get the IP address of Ambassador, run the following commands:
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
     You can watch the status of by running 'kubectl get svc -w  --namespace seldon ambassador'

  On GKE/Azure:
  export SERVICE_IP=$(kubectl get svc --namespace seldon ambassador -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

  On AWS:
  export SERVICE_IP=$(kubectl get svc --namespace seldon ambassador -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

  echo http://$SERVICE_IP:

[59]:
!kubectl rollout status deployment.apps/ambassador
Waiting for deployment "ambassador" rollout to finish: 0 of 1 updated replicas are available...
deployment "ambassador" successfully rolled out

Port Forward to Ambassador

kubectl port-forward $(kubectl get pods -n seldon -l app.kubernetes.io/name=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8003:8080

Serve Single Model

[60]:
!helm install ../helm-charts/seldon-single-model --name mymodel
NAME:   mymodel
LAST DEPLOYED: Wed Aug 14 09:21:13 2019
NAMESPACE: seldon
STATUS: DEPLOYED

RESOURCES:
==> v1alpha2/SeldonDeployment
NAME     AGE
mymodel  0s


[10]:
!helm template ../helm-charts/seldon-single-model | pygmentize -l json
---
# Source: seldon-single-model/templates/model.json
{
    "apiVersion": "machinelearning.seldon.io/v1alpha2",
    "kind": "SeldonDeployment",
    "metadata": {
        "labels": {"app":"seldon"},
        "name": "release-name"
    },
    "spec": {
        "annotations":{"seldon.io/engine-log-message-type":"seldon.message.pair","seldon.io/engine-log-messages-externally":"false","seldon.io/engine-log-requests":"false","seldon.io/engine-log-responses":"false","seldon.io/headless-svc":"false"},
        "name": "release-name",
        "predictors": [
            {
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/mock_classifier:1.0",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier",
                                "resources": {"requests":{"memory":"1Mi"}},
                                "env": [
{
"name": "LOG_LEVEL",
"value": "INFO"
},
]
                            }
                        ],
                        "terminationGracePeriodSeconds": 1
                    }}
                                  ],
                "graph":
                {
                    "children": [],
                    "name": "classifier",
                    "type": "MODEL",
                    "endpoint": {
                        "type": "REST"
                    }},
                "svcOrchSpec": {
                "resources": {"requests":{"cpu":"0.1"}},
"env": [
]
                },
                "name": "release-name",
                "replicas": 1,
            "labels": {"fluentd":"true","version":"v1"}
            }
        ]
    }
}
[61]:
!kubectl rollout status deploy/mymodel-mymodel-7cd068f
deployment "mymodel-mymodel-7cd068f" successfully rolled out

Get predictions

[62]:
from seldon_core.seldon_client import SeldonClient
sc = SeldonClient(deployment_name="mymodel",namespace="seldon",gateway_endpoint="localhost:8003",gateway="ambassador")

REST Request

[63]:
r = sc.predict(transport="rest")
print(r)
Success:True message:
Request:
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.7294723174563513
  }
}

Response:
meta {
  puid: "m3oedepvj7dqrd8r25nh5juakl"
  requestPath {
    key: "classifier"
    value: "seldonio/mock_classifier:1.0"
  }
}
data {
  names: "proba"
  tensor {
    shape: 1
    shape: 1
    values: 0.10090634928875347
  }
}

gRPC Request

[64]:
r = sc.predict(transport="grpc")
print(r)
Success:True message:
Request:
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.9816337176739471
  }
}

Response:
meta {
  puid: "864t9vhkarg82mc06kbl1c48bl"
  requestPath {
    key: "classifier"
    value: "seldonio/mock_classifier:1.0"
  }
}
data {
  names: "proba"
  tensor {
    shape: 1
    shape: 1
    values: 0.1261945545073467
  }
}

[65]:
!helm delete mymodel --purge
release "mymodel" deleted

Serve AB Test

[66]:
!helm install ../helm-charts/seldon-abtest --name myabtest
NAME:   myabtest
LAST DEPLOYED: Wed Aug 14 09:23:40 2019
NAMESPACE: seldon
STATUS: DEPLOYED

RESOURCES:
==> v1alpha2/SeldonDeployment
NAME      AGE
myabtest  1s


[19]:
!helm template ../helm-charts/seldon-abtest | pygmentize -l json
---
# Source: seldon-abtest/templates/ab_test_1pod.json


---
# Source: seldon-abtest/templates/ab_test_2pods.json

{
    "apiVersion": "machinelearning.seldon.io/v1alpha2",
    "kind": "SeldonDeployment",
    "metadata": {
        "labels": {
            "app": "seldon"
        },
        "name": "release-name"
    },
    "spec": {
        "name": "release-name",
        "predictors": [
            {
                "name": "release-name",
                "replicas": 1,
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/mock_classifier:1.0",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier-1",
                                "resources": {
                                    "requests": {
                                        "memory": "1Mi"
                                    }
                                }
                            }],
                        "terminationGracePeriodSeconds": 20
                    }},
                {
                    "metadata":{
                        "labels":{
                            "version":"v2"
                        }
                    },
                        "spec":{
                            "containers":[
                            {
                                "image": "seldonio/mock_classifier:1.0",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier-2",
                                "resources": {
                                    "requests": {
                                        "memory": "1Mi"
                                    }
                                }
                            }
                        ],
                        "terminationGracePeriodSeconds": 20
                                   }
                                   }],
                "graph": {
                    "name": "release-name",
                    "endpoint":{},
                    "implementation":"RANDOM_ABTEST",
                    "parameters": [
                        {
                            "name":"ratioA",
                            "value":"0.5",
                            "type":"FLOAT"
                        }
                    ],
                    "children": [
                        {
                            "name": "classifier-1",
                            "endpoint":{
                                "type":"REST"
                            },
                            "type":"MODEL",
                            "children":[]
                        },
                        {
                            "name": "classifier-2",
                            "endpoint":{
                                "type":"REST"
                            },
                            "type":"MODEL",
                            "children":[]
                        }
                    ]
                }
            }
        ]
    }
}



[67]:
!kubectl rollout status deploy/myabtest-myabtest-41de5b8
!kubectl rollout status deploy/myabtest-myabtest-df66c5c
deployment "myabtest-myabtest-41de5b8" successfully rolled out
deployment "myabtest-myabtest-df66c5c" successfully rolled out

Get predictions

[68]:
from seldon_core.seldon_client import SeldonClient
sc = SeldonClient(deployment_name="myabtest",namespace="seldon",gateway_endpoint="localhost:8003",gateway="ambassador")

REST Request

[69]:
r = sc.predict(transport="rest")
print(r)
Success:True message:
Request:
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.7243037145484317
  }
}

Response:
meta {
  puid: "jkbbad2ck5572fuq3m97hf1jnb"
  routing {
    key: "myabtest"
    value: 1
  }
  requestPath {
    key: "classifier-2"
    value: "seldonio/mock_classifier:1.0"
  }
  requestPath {
    key: "myabtest"
    value: ""
  }
}
data {
  names: "proba"
  tensor {
    shape: 1
    shape: 1
    values: 0.10043839793672894
  }
}

gRPC Request

[76]:
r = sc.predict(transport="grpc")
print(r)
Success:True message:
Request:
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.11690007088816501
  }
}

Response:
meta {
  puid: "4h2r8fn7hcu1o3lldhlrus1rns"
  routing {
    key: "myabtest"
    value: 0
  }
  requestPath {
    key: "classifier-1"
    value: "seldonio/mock_classifier:1.0"
  }
  requestPath {
    key: "myabtest"
    value: ""
  }
}
data {
  names: "proba"
  tensor {
    shape: 1
    shape: 1
    values: 0.057336789926229086
  }
}

[77]:
!helm delete myabtest --purge
release "myabtest" deleted

Serve Multi-Armed Bandit

[78]:
!helm install ../helm-charts/seldon-mab --name mymab
NAME:   mymab
LAST DEPLOYED: Wed Aug 14 09:24:49 2019
NAMESPACE: seldon
STATUS: DEPLOYED

RESOURCES:
==> v1alpha2/SeldonDeployment
NAME   AGE
mymab  0s


[96]:
!helm template ../helm-charts/seldon-mab | pygmentize -l json
---
# Source: seldon-mab/templates/mab.json
{
    "apiVersion": "machinelearning.seldon.io/v1alpha2",
    "kind": "SeldonDeployment",
    "metadata": {
                "labels": {"app":"seldon"},
                "name": "release-name"
    },
    "spec": {
        "name": "release-name",
        "predictors": [
            {
                "name": "release-name",
                "replicas": 1,
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/mock_classifier:1.0",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier-1",
                                "resources": {
                                    "requests": {
                                        "memory": "1Mi"
                                    }
                                }
                            }],
                        "terminationGracePeriodSeconds": 20
                    }},
                {
                        "spec":{
                            "containers":[
                            {
                                "image": "seldonio/mock_classifier:1.0",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier-2",
                                "resources": {
                                    "requests": {
                                        "memory": "1Mi"
                                    }
                                }
                            }
                        ],
                        "terminationGracePeriodSeconds": 20
                        }
                },
                {
                    "spec":{
                        "containers": [{
                            "image": "seldonio/mab_epsilon_greedy:1.1",
                            "name": "eg-router"
                        }],
                        "terminationGracePeriodSeconds": 20
                    }}
                ],
                "graph": {
                    "name": "eg-router",
                    "type":"ROUTER",
                    "parameters": [
                        {
                            "name": "n_branches",
                            "value": "2",
                            "type": "INT"
                        },
                        {
                            "name": "epsilon",
                            "value": "0.2",
                            "type": "FLOAT"
                        },
                        {
                            "name": "verbose",
                            "value": "1",
                            "type": "BOOL"
                        }
                    ],
                    "children": [
                        {
                            "name": "classifier-1",
                            "endpoint":{
                                "type":"REST"
                            },
                            "type":"MODEL",
                            "children":[]
                        },
                        {
                            "name": "classifier-2",
                            "endpoint":{
                                "type":"REST"
                            },
                            "type":"MODEL",
                            "children":[]
                        }
                    ]
                },
                "svcOrchSpec": {
                "resources": {"requests":{"cpu":"0.1"}},
"env": [
{
"name": "SELDON_LOG_MESSAGES_EXTERNALLY",
"value": "false"
},
{
"name": "SELDON_LOG_MESSAGE_TYPE",
"value": "seldon.message.pair"
},
{
"name": "SELDON_LOG_REQUESTS",
"value": "false"
},
{
"name": "SELDON_LOG_RESPONSES",
"value": "false"
},
]
},
                "labels": {"fluentd":"true","version":"v1"}
            }
        ]
    }
}



[79]:
!kubectl rollout status deploy/mymab-mymab-41de5b8
!kubectl rollout status deploy/mymab-mymab-b8038b2
!kubectl rollout status deploy/mymab-mymab-df66c5c
deployment "mymab-mymab-41de5b8" successfully rolled out
deployment "mymab-mymab-b8038b2" successfully rolled out
deployment "mymab-mymab-df66c5c" successfully rolled out

Get predictions

[80]:
from seldon_core.seldon_client import SeldonClient
sc = SeldonClient(deployment_name="mymab",namespace="seldon",gateway_endpoint="localhost:8003",gateway="ambassador")

REST Request

[81]:
r = sc.predict(transport="rest")
print(r)
Success:True message:
Request:
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.41418976743464186
  }
}

Response:
meta {
  puid: "930tive9mqkli5dk6d2odnssu8"
  routing {
    key: "eg-router"
    value: 1
  }
  requestPath {
    key: "classifier-2"
    value: "seldonio/mock_classifier:1.0"
  }
  requestPath {
    key: "eg-router"
    value: "seldonio/mab_epsilon_greedy:1.1"
  }
}
data {
  names: "proba"
  tensor {
    shape: 1
    shape: 1
    values: 0.07568472249519131
  }
}

gRPC Request

[85]:
r = sc.predict(transport="grpc")
print(r)
Success:True message:
Request:
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.5241212703444827
  }
}

Response:
meta {
  puid: "p92hk3957rl0uieuss0ucia2bg"
  routing {
    key: "eg-router"
    value: 0
  }
  requestPath {
    key: "classifier-1"
    value: "seldonio/mock_classifier:1.0"
  }
  requestPath {
    key: "eg-router"
    value: "seldonio/mab_epsilon_greedy:1.1"
  }
}
data {
  names: "proba"
  tensor {
    shape: 1
    shape: 1
    values: 0.08374291648417304
  }
}

[52]:
!helm delete mymab --purge
release "mymab" deleted
[ ]: