This page was generated from notebooks/istio_example.ipynb.

Example Seldon Core Deployments using Helm

Setup Cluster and Ingress

Use the setup notebook to Setup Cluster with Istio Ingress. Instructions also online.

Configure Istio

For this example we will create the default istio gateway for seldon which needs to be called seldon-gateway. You can supply your own gateway by adding to your SeldonDeployments resources the annotation seldon.io/istio-gateway with values the name of your istio gateway.

Create a gateway for our istio-ingress

[52]:
!kubectl create -f resources/seldon-gateway.yaml
gateway.networking.istio.io/seldon-gateway created

Label our namespace so istio creates sidecars

[53]:
!kubectl label namespace seldon istio-injection=enabled
namespace/seldon labeled

If you are using Minikube for your Kubernetes cluster you will need to run as root in a separte terminal:

minikube tunnel

This will allow a LoadBalancer to be simulated on your local machine.

[54]:
INGRESS_HOST=!kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
INGRESS_PORT=!kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].port}'
ISTIO_GATEWAY=INGRESS_HOST[0]+":"+INGRESS_PORT[0]
[55]:
ISTIO_GATEWAY
[55]:
'34.68.6.137:80'

Start Seldon Core

Use the setup notebook to Install Seldon Core with Istio Ingress. Instructions also online.

Serve Single Model

[56]:
!helm install mymodel ../helm-charts/seldon-single-model
NAME: mymodel
LAST DEPLOYED: Mon Dec  2 14:51:14 2019
NAMESPACE: seldon
STATUS: deployed
REVISION: 1
TEST SUITE: None
[57]:
!helm template ../helm-charts/seldon-single-model | pygmentize -l json
---
# Source: seldon-single-model/templates/model.json
{
    "apiVersion": "machinelearning.seldon.io/v1alpha2",
    "kind": "SeldonDeployment",
    "metadata": {
        "labels": {"app":"seldon"},
        "name": "RELEASE-NAME"
    },
    "spec": {
        "annotations":{"seldon.io/engine-log-message-type":"seldon.message.pair","seldon.io/engine-log-messages-externally":"false","seldon.io/engine-log-requests":"false","seldon.io/engine-log-responses":"false","seldon.io/headless-svc":"false"},
        "name": "RELEASE-NAME",
        "predictors": [
            {
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/mock_classifier:1.0",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier",
                                "resources": {"requests":{"memory":"1Mi"}},
                                "env": [
{
"name": "LOG_LEVEL",
"value": "INFO"
},
]
                            }
                        ],
                        "terminationGracePeriodSeconds": 1
                    }}
                                  ],
                "graph":
                {
                    "children": [],
                    "name": "classifier",
                    "type": "MODEL",
                    "endpoint": {
                        "type": "REST"
                    }},
                "svcOrchSpec": {
                "resources": {"requests":{"cpu":"0.1"}},
"env": [
]
                },
                "name": "RELEASE-NAME",
                "replicas": 1,
            "labels": {"fluentd":"true","version":"v1"}
            }
        ]
    }
}
[58]:
!kubectl rollout status deploy/mymodel-mymodel-7cd068f
Waiting for deployment "mymodel-mymodel-7cd068f" rollout to finish: 0 of 1 updated replicas are available...
deployment "mymodel-mymodel-7cd068f" successfully rolled out

Get predictions

[59]:
from seldon_core.seldon_client import SeldonClient
sc = SeldonClient(deployment_name="mymodel",namespace="seldon",gateway_endpoint=ISTIO_GATEWAY)

REST Request

[60]:
r = sc.predict(gateway="istio",transport="rest")
print(r)
Success:True message:
Request:
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.6791270322000476
  }
}

Response:
meta {
  puid: "gs20ma826fgis83pak76p8gvf2"
  requestPath {
    key: "classifier"
    value: "seldonio/mock_classifier:1.0"
  }
}
data {
  names: "proba"
  tensor {
    shape: 1
    shape: 1
    values: 0.09642970310262379
  }
}

gRPC Request

[61]:
r = sc.predict(gateway="istio",transport="grpc")
print(r)
Success:True message:
Request:
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.8889126802531476
  }
}

Response:
meta {
  puid: "tjlfo3ngj11t59fde579jpfbhc"
  requestPath {
    key: "classifier"
    value: "seldonio/mock_classifier:1.0"
  }
}
data {
  names: "proba"
  tensor {
    shape: 1
    shape: 1
    values: 0.11631958945402368
  }
}

[62]:
!helm delete mymodel
release "mymodel" uninstalled

Serve AB Test

[12]:
!helm install myabtest ../helm-charts/seldon-abtest
NAME: myabtest
LAST DEPLOYED: Mon Dec  2 14:19:15 2019
NAMESPACE: seldon
STATUS: deployed
REVISION: 1
TEST SUITE: None
[13]:
!helm template ../helm-charts/seldon-abtest | pygmentize -l json
---
# Source: seldon-abtest/templates/ab_test_2pods.json
{
    "apiVersion": "machinelearning.seldon.io/v1alpha2",
    "kind": "SeldonDeployment",
    "metadata": {
        "labels": {
            "app": "seldon"
        },
        "name": "RELEASE-NAME"
    },
    "spec": {
        "name": "RELEASE-NAME",
        "predictors": [
            {
                "name": "RELEASE-NAME",
                "replicas": 1,
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/mock_classifier:1.0",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier-1",
                                "resources": {
                                    "requests": {
                                        "memory": "1Mi"
                                    }
                                }
                            }],
                        "terminationGracePeriodSeconds": 20
                    }},
                {
                    "metadata":{
                        "labels":{
                            "version":"v2"
                        }
                    },
                        "spec":{
                            "containers":[
                            {
                                "image": "seldonio/mock_classifier:1.0",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier-2",
                                "resources": {
                                    "requests": {
                                        "memory": "1Mi"
                                    }
                                }
                            }
                        ],
                        "terminationGracePeriodSeconds": 20
                                   }
                                   }],
                "graph": {
                    "name": "RELEASE-NAME",
                    "endpoint":{},
                    "implementation":"RANDOM_ABTEST",
                    "parameters": [
                        {
                            "name":"ratioA",
                            "value":"0.5",
                            "type":"FLOAT"
                        }
                    ],
                    "children": [
                        {
                            "name": "classifier-1",
                            "endpoint":{
                                "type":"REST"
                            },
                            "type":"MODEL",
                            "children":[]
                        },
                        {
                            "name": "classifier-2",
                            "endpoint":{
                                "type":"REST"
                            },
                            "type":"MODEL",
                            "children":[]
                        }
                    ]
                }
            }
        ]
    }
}
[14]:
!kubectl rollout status deploy/myabtest-myabtest-41de5b8
!kubectl rollout status deploy/myabtest-myabtest-df66c5c
Waiting for deployment "myabtest-myabtest-41de5b8" rollout to finish: 0 of 1 updated replicas are available...
deployment "myabtest-myabtest-41de5b8" successfully rolled out
deployment "myabtest-myabtest-df66c5c" successfully rolled out

Get predictions

[15]:
from seldon_core.seldon_client import SeldonClient
sc = SeldonClient(deployment_name="myabtest",namespace="seldon",gateway_endpoint=ISTIO_GATEWAY)

REST Request

[16]:
r = sc.predict(gateway="istio",transport="rest")
print(r)
Success:True message:
Request:
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.3288782140746789
  }
}

Response:
meta {
  puid: "lf4cnru26oag43uj3frj03k3hi"
  routing {
    key: "myabtest"
    value: 1
  }
  requestPath {
    key: "classifier-2"
    value: "seldonio/mock_classifier:1.0"
  }
  requestPath {
    key: "myabtest"
    value: ""
  }
}
data {
  names: "proba"
  tensor {
    shape: 1
    shape: 1
    values: 0.06992848184490845
  }
}

gRPC Request

[17]:
r = sc.predict(gateway="istio",transport="grpc")
print(r)
Success:True message:
Request:
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.13958830227813712
  }
}

Response:
meta {
  puid: "i0hd016h1trpnsl72k4srciurl"
  routing {
    key: "myabtest"
    value: 0
  }
  requestPath {
    key: "classifier-1"
    value: "seldonio/mock_classifier:1.0"
  }
  requestPath {
    key: "myabtest"
    value: ""
  }
}
data {
  names: "proba"
  tensor {
    shape: 1
    shape: 1
    values: 0.058575459683195934
  }
}

[18]:
!helm delete myabtest
release "myabtest" uninstalled

Serve Multi-Armed Bandit

[40]:
!helm install mymab ../helm-charts/seldon-mab
NAME: mymab
LAST DEPLOYED: Mon Dec  2 14:28:11 2019
NAMESPACE: seldon
STATUS: deployed
REVISION: 1
TEST SUITE: None
[41]:
!helm template ../helm-charts/seldon-mab | pygmentize -l json
---
# Source: seldon-mab/templates/mab.json
{
    "apiVersion": "machinelearning.seldon.io/v1alpha2",
    "kind": "SeldonDeployment",
    "metadata": {
                "labels": {"app":"seldon"},
                "name": "RELEASE-NAME"
    },
    "spec": {
        "name": "RELEASE-NAME",
        "predictors": [
            {
                "name": "mymab",
                "replicas": 1,
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/mock_classifier:1.0",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier-1",
                                "resources": {
                                    "requests": {
                                        "memory": "1Mi"
                                    }
                                }
                            }],
                        "terminationGracePeriodSeconds": 20
                    }},
                {
                        "spec":{
                            "containers":[
                            {
                                "image": "seldonio/mock_classifier:1.0",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier-2",
                                "resources": {
                                    "requests": {
                                        "memory": "1Mi"
                                    }
                                }
                            }
                        ],
                        "terminationGracePeriodSeconds": 20
                        }
                },
                {
                    "spec":{
                        "containers": [{
                            "image": "seldonio/mab_epsilon_greedy:1.1",
                            "name": "eg-router"
                        }],
                        "terminationGracePeriodSeconds": 20
                    }}
                ],
                "graph": {
                    "name": "eg-router",
                    "type":"ROUTER",
                    "parameters": [
                        {
                            "name": "n_branches",
                            "value": "2",
                            "type": "INT"
                        },
                        {
                            "name": "epsilon",
                            "value": "0.2",
                            "type": "FLOAT"
                        },
                        {
                            "name": "verbose",
                            "value": "1",
                            "type": "BOOL"
                        }
                    ],
                    "children": [
                        {
                            "name": "classifier-1",
                            "endpoint":{
                                "type":"REST"
                            },
                            "type":"MODEL",
                            "children":[]
                        },
                        {
                            "name": "classifier-2",
                            "endpoint":{
                                "type":"REST"
                            },
                            "type":"MODEL",
                            "children":[]
                        }
                    ]
                },
                "svcOrchSpec": {
                "resources": {"requests":{"cpu":"0.1"}},
"env": [
{
"name": "SELDON_LOG_MESSAGES_EXTERNALLY",
"value": "false"
},
{
"name": "SELDON_LOG_MESSAGE_TYPE",
"value": "seldon.message.pair"
},
{
"name": "SELDON_LOG_REQUESTS",
"value": "false"
},
{
"name": "SELDON_LOG_RESPONSES",
"value": "false"
},
]
},
                "labels": {"fluentd":"true","version":"v1"}
            }
        ]
    }
}
[42]:
!kubectl rollout status deploy/mymab-mymab-41de5b8
!kubectl rollout status deploy/mymab-mymab-b8038b2
!kubectl rollout status deploy/mymab-mymab-df66c5c
Waiting for deployment "mymab-mymab-41de5b8" rollout to finish: 0 of 1 updated replicas are available...
deployment "mymab-mymab-41de5b8" successfully rolled out
deployment "mymab-mymab-b8038b2" successfully rolled out
deployment "mymab-mymab-df66c5c" successfully rolled out

Get predictions

[43]:
from seldon_core.seldon_client import SeldonClient
sc = SeldonClient(deployment_name="mymab",namespace="seldon",gateway_endpoint=ISTIO_GATEWAY)

REST Request

[44]:
r = sc.predict(gateway="istio",transport="rest")
print(r)
Success:True message:
Request:
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.5374804593634502
  }
}

Response:
meta {
  puid: "poru4c1m2o4ho4u1cmvcvfkgl9"
  routing {
    key: "eg-router"
    value: 0
  }
  requestPath {
    key: "classifier-1"
    value: "seldonio/mock_classifier:1.0"
  }
  requestPath {
    key: "eg-router"
    value: "seldonio/mab_epsilon_greedy:1.1"
  }
}
data {
  names: "proba"
  tensor {
    shape: 1
    shape: 1
    values: 0.08477368421978464
  }
}

gRPC Request

[24]:
r = sc.predict(gateway="istio",transport="grpc")
print(r)
Success:True message:
Request:
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.6719798732423108
  }
}

Response:
meta {
  puid: "u1k779usglt5qansr4sbujia24"
  routing {
    key: "eg-router"
    value: 0
  }
  requestPath {
    key: "classifier-1"
    value: "seldonio/mock_classifier:1.0"
  }
  requestPath {
    key: "eg-router"
    value: "seldonio/mab_epsilon_greedy:1.1"
  }
}
data {
  names: "proba"
  tensor {
    shape: 1
    shape: 1
    values: 0.09580875757175003
  }
}

[45]:
!helm delete mymab
release "mymab" uninstalled

Serve with Shadow

[63]:
!pygmentize ./resources/istio_shadow.yaml
kind: SeldonDeployment
apiVersion: machinelearning.seldon.io/v1alpha2
metadata:
  name: iris
spec:
  name: iris
  predictors:
    - name: default
      graph:
        name: iris-default
        implementation: SKLEARN_SERVER
        modelUri: gs://seldon-models/sklearn/iris
      replicas: 1
    - name: shadow
      graph:
        name: iris-shadow
        implementation: SKLEARN_SERVER
        modelUri: gs://seldon-models/sklearn/iris
      replicas: 1
      shadow: true
[64]:
!kubectl apply -f ./resources/istio_shadow.yaml
seldondeployment.machinelearning.seldon.io/iris created
[65]:
!kubectl rollout status deploy/iris-default-54fcd84
!kubectl rollout status deploy/iris-shadow-446a1b8
Waiting for deployment "iris-default-54fcd84" rollout to finish: 0 of 1 updated replicas are available...
deployment "iris-default-54fcd84" successfully rolled out
Waiting for deployment "iris-shadow-446a1b8" rollout to finish: 0 of 1 updated replicas are available...
deployment "iris-shadow-446a1b8" successfully rolled out
[66]:
from seldon_core.seldon_client import SeldonClient
sc = SeldonClient(deployment_name="sklearn",namespace="seldon",gateway_endpoint=ISTIO_GATEWAY)
[71]:
r = sc.predict(gateway="istio",transport="rest",shape=(1,4))
print(r)
Success:False message:404:Not Found
Request:
data {
  tensor {
    shape: 1
    shape: 4
    values: 0.276288032414445
    values: 0.0985589669202912
    values: 0.25419907181414625
    values: 0.6812151048957957
  }
}

Response:
None
[51]:
!kubectl delete -f ./resources/istio_shadow.yaml
seldondeployment.machinelearning.seldon.io "iris" deleted
[ ]: