This page was generated from notebooks/backwards_compatability.ipynb.

Backwards Compatability Examples with Different Protocols

Prerequisites

  • A kubernetes cluster with kubectl configured

  • curl

  • grpcurl

  • pygmentize

Setup Seldon Core

Use the setup notebook to Setup Cluster to setup Seldon Core with an ingress - either Ambassador or Istio.

Then port-forward to that ingress on localhost:8003 in a separate terminal either with:

  • Ambassador: kubectl port-forward $(kubectl get pods -n seldon -l app.kubernetes.io/name=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8003:8080

  • Istio: kubectl port-forward $(kubectl get pods -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}') -n istio-system 8003:80

[1]:
!kubectl create namespace seldon
Error from server (AlreadyExists): namespaces "seldon" already exists
[2]:
!kubectl config set-context $(kubectl config current-context) --namespace=seldon
Context "kind-kind" modified.
[3]:
import json
import time
[4]:
from IPython.core.magic import register_line_cell_magic

@register_line_cell_magic
def writetemplate(line, cell):
    with open(line, 'w') as f:
        f.write(cell.format(**globals()))
[5]:
VERSION=!cat ../version.txt
VERSION=VERSION[0]
VERSION
[5]:
'1.5.0-dev'

Model with Old REST Wrapper Upgraded

We will deploy a REST model that uses the SELDON Protocol namely by specifying the attribute protocol: seldon

[26]:
%%writetemplate resources/model_seldon.yaml
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: example-seldon
spec:
  protocol: seldon
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: seldonio/mock_classifier_rest:1.4.0
          name: classifier
    graph:
      name: classifier
      type: MODEL
    name: model
    replicas: 1
[27]:
!kubectl apply -f resources/model_seldon.yaml
seldondeployment.machinelearning.seldon.io/example-seldon created
[28]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=example-seldon -o jsonpath='{.items[0].metadata.name}')
Waiting for deployment "example-seldon-model-0-classifier" rollout to finish: 0 of 1 updated replicas are available...
deployment "example-seldon-model-0-classifier" successfully rolled out
[29]:
for i in range(60):
    state=!kubectl get sdep example-seldon -o jsonpath='{.status.state}'
    state=state[0]
    print(state)
    if state=="Available":
        break
    time.sleep(1)
assert(state=="Available")
Available
[30]:
X=!curl -s -d '{"data": {"ndarray":[[1.0, 2.0, 5.0]]}}' \
   -X POST http://localhost:8003/seldon/seldon/example-seldon/api/v1.0/predictions \
   -H "Content-Type: application/json"
d=json.loads(X[0])
print(d)
assert(d["data"]["ndarray"][0][0] > 0.4)
{'data': {'names': ['proba'], 'ndarray': [[0.43782349911420193]]}, 'meta': {}}
[31]:
%%writetemplate resources/model_seldon.yaml
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: example-seldon
spec:
  protocol: seldon
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: seldonio/mock_classifier:{VERSION}
          name: classifier
    graph:
      name: classifier
      type: MODEL
    name: model
    replicas: 1
[32]:
!kubectl apply -f resources/model_seldon.yaml
seldondeployment.machinelearning.seldon.io/example-seldon configured
[33]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=example-seldon -o jsonpath='{.items[0].metadata.name}')
Waiting for deployment "example-seldon-model-0-classifier" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "example-seldon-model-0-classifier" rollout to finish: 1 old replicas are pending termination...
deployment "example-seldon-model-0-classifier" successfully rolled out
[34]:
for i in range(60):
    state=!kubectl get sdep example-seldon -o jsonpath='{.status.state}'
    state=state[0]
    print(state)
    if state=="Available":
        break
    time.sleep(1)
assert(state=="Available")
Available
[35]:
X=!curl -s -d '{"data": {"ndarray":[[1.0, 2.0, 5.0]]}}' \
   -X POST http://localhost:8003/seldon/seldon/example-seldon/api/v1.0/predictions \
   -H "Content-Type: application/json"
d=json.loads(X[0])
print(d)
assert(d["data"]["ndarray"][0][0] > 0.4)
{'data': {'names': ['proba'], 'ndarray': [[0.43782349911420193]]}, 'meta': {}}
[36]:
X=!cd ../executor/proto && grpcurl -d '{"data":{"ndarray":[[1.0,2.0,5.0]]}}' \
         -rpc-header seldon:example-seldon -rpc-header namespace:seldon \
         -plaintext \
         -proto ./prediction.proto  0.0.0.0:8003 seldon.protos.Seldon/Predict
d=json.loads("".join(X))
print(d)
assert(d["data"]["ndarray"][0][0] > 0.4)
{'meta': {}, 'data': {'names': ['proba'], 'ndarray': [[0.43782349911420193]]}}
[37]:
!kubectl delete -f resources/model_seldon.yaml
seldondeployment.machinelearning.seldon.io "example-seldon" deleted

Model with Old GRPC Wrapper Upgraded

We will deploy a gRPC model that uses the SELDON Protocol namely by specifying the attribute protocol: seldon

[19]:
%%writefile resources/model_seldon_grpc.yaml
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: grpc-seldon
spec:
  name: grpcseldon
  protocol: seldon
  transport: grpc
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: seldonio/mock_classifier_grpc:1.3
          name: classifier
    graph:
      name: classifier
      type: MODEL
      endpoint:
        type: GRPC
    name: model
    replicas: 1
Overwriting resources/model_seldon_grpc.yaml
[20]:
!kubectl apply -f resources/model_seldon_grpc.yaml
seldondeployment.machinelearning.seldon.io/grpc-seldon created
[21]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=grpc-seldon -o jsonpath='{.items[0].metadata.name}')
Waiting for deployment "grpc-seldon-model-0-classifier" rollout to finish: 0 of 1 updated replicas are available...
deployment "grpc-seldon-model-0-classifier" successfully rolled out
[23]:
for i in range(60):
    state=!kubectl get sdep grpc-seldon -o jsonpath='{.status.state}'
    state=state[0]
    print(state)
    if state=="Available":
        break
    time.sleep(1)
assert(state=="Available")
Available
[24]:
X=!cd ../executor/proto && grpcurl -d '{"data":{"ndarray":[[1.0,2.0,5.0,6.0]]}}' \
         -rpc-header seldon:grpc-seldon -rpc-header namespace:seldon \
         -plaintext \
         -proto ./prediction.proto  0.0.0.0:8003 seldon.protos.Seldon/Predict
d=json.loads("".join(X))
print(d)
{'meta': {}, 'data': {'names': ['proba'], 'ndarray': [[0.6418340450887311]]}}
[25]:
!kubectl delete -f resources/model_seldon_grpc.yaml
seldondeployment.machinelearning.seldon.io "grpc-seldon" deleted

Old Operator and Model Upgraded

[1]:
!helm delete seldon seldon-core-operator \
    --namespace seldon-system
release "seldon" uninstalled
Error: uninstall: Release not loaded: seldon-core-operator: release: not found
[2]:
!helm install seldon seldon-core-operator \
    --repo https://storage.googleapis.com/seldon-charts \
    --version 1.4.0 \
    --namespace seldon-system \
    --wait
NAME: seldon
LAST DEPLOYED: Mon Nov  9 08:57:55 2020
NAMESPACE: seldon-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
[9]:
%%writetemplate resources/model_seldon.yaml
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: example-seldon
spec:
  protocol: seldon
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: seldonio/mock_classifier_rest:1.4.0
          name: classifier
    graph:
      name: classifier
      type: MODEL
    name: model
    replicas: 1
[10]:
%%writefile ../servers/sklearnserver/samples/iris.yaml
apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  name: sklearn
spec:
  name: iris
  predictors:
  - graph:
      children: []
      implementation: SKLEARN_SERVER
      modelUri: gs://seldon-models/sklearn/iris
      name: classifier
    name: default
    replicas: 1
    svcOrchSpec:
      env:
      - name: SELDON_LOG_LEVEL
        value: DEBUG
Overwriting ../servers/sklearnserver/samples/iris.yaml
[11]:
!kubectl apply -f resources/model_seldon.yaml
seldondeployment.machinelearning.seldon.io/example-seldon created
[12]:
!kubectl apply -f ../servers/sklearnserver/samples/iris.yaml
seldondeployment.machinelearning.seldon.io/sklearn created
[13]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=example-seldon -o jsonpath='{.items[0].metadata.name}')
Waiting for deployment "example-seldon-model-0-classifier" rollout to finish: 0 of 1 updated replicas are available...
deployment "example-seldon-model-0-classifier" successfully rolled out
[14]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=sklearn -o jsonpath='{.items[0].metadata.name}')
deployment "sklearn-default-0-classifier" successfully rolled out
[15]:
for i in range(60):
    state=!kubectl get sdep example-seldon -o jsonpath='{.status.state}'
    state=state[0]
    print(state)
    if state=="Available":
        break
    time.sleep(1)
assert(state=="Available")
Available
[16]:
X=!curl -s -d '{"data": {"ndarray":[[1.0, 2.0, 5.0]]}}' \
   -X POST http://localhost:8003/seldon/seldon/example-seldon/api/v1.0/predictions \
   -H "Content-Type: application/json"
d=json.loads(X[0])
print(d)
assert(d["data"]["ndarray"][0][0] > 0.4)
{'data': {'names': ['proba'], 'ndarray': [[0.43782349911420193]]}, 'meta': {}}
[17]:
X=!curl -s -d '{"data": {"ndarray":[[1.0, 2.0, 5.0, 6.0]]}}' \
   -X POST http://localhost:8003/seldon/seldon/sklearn/api/v1.0/predictions \
   -H "Content-Type: application/json"
d=json.loads(X[0])
print(d)
{'data': {'names': ['t:0', 't:1', 't:2'], 'ndarray': [[9.912315378486718e-07, 0.0007015931307743852, 0.9992974156376878]]}, 'meta': {}}
[18]:
!helm upgrade seldon \
    ../helm-charts/seldon-core-operator \
    --namespace seldon-system \
    --wait
Release "seldon" has been upgraded. Happy Helming!
NAME: seldon
LAST DEPLOYED: Mon Nov  9 09:06:11 2020
NAMESPACE: seldon-system
STATUS: deployed
REVISION: 2
TEST SUITE: None
[19]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=example-seldon -o jsonpath='{.items[0].metadata.name}')
deployment "example-seldon-model-0-classifier" successfully rolled out
[20]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=sklearn -o jsonpath='{.items[0].metadata.name}')
deployment "sklearn-default-0-classifier" successfully rolled out
[21]:
for i in range(60):
    state=!kubectl get sdep example-seldon -o jsonpath='{.status.state}'
    state=state[0]
    print(state)
    if state=="Available":
        break
    time.sleep(1)
assert(state=="Available")
Available

Only REST calls will be available as image is still old python wrapper

[22]:
X=!curl -s -d '{"data": {"ndarray":[[1.0, 2.0, 5.0]]}}' \
   -X POST http://localhost:8003/seldon/seldon/example-seldon/api/v1.0/predictions \
   -H "Content-Type: application/json"
d=json.loads(X[0])
print(d)
assert(d["data"]["ndarray"][0][0] > 0.4)
{'data': {'names': ['proba'], 'ndarray': [[0.43782349911420193]]}, 'meta': {}}

Rest and gRPC calls will work with new server as image will have been updated.

[23]:
X=!curl -s -d '{"data": {"ndarray":[[1.0, 2.0, 5.0, 6.0]]}}' \
   -X POST http://localhost:8003/seldon/seldon/sklearn/api/v1.0/predictions \
   -H "Content-Type: application/json"
d=json.loads(X[0])
print(d)
{'data': {'names': ['t:0', 't:1', 't:2'], 'ndarray': [[9.912315378486718e-07, 0.0007015931307743852, 0.9992974156376878]]}, 'meta': {}}
[24]:
X=!cd ../executor/proto && grpcurl -d '{"data":{"ndarray":[[1.0,2.0,5.0,6.0]]}}' \
         -rpc-header seldon:sklearn -rpc-header namespace:seldon \
         -plaintext \
         -proto ./prediction.proto  0.0.0.0:8003 seldon.protos.Seldon/Predict
d=json.loads("".join(X))
print(d)
{'meta': {}, 'data': {'names': ['t:0', 't:1', 't:2'], 'ndarray': [[9.912315378486718e-07, 0.0007015931307743852, 0.9992974156376878]]}}
[25]:
%%writetemplate resources/model_seldon.yaml
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: example-seldon
spec:
  protocol: seldon
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: seldonio/mock_classifier:{VERSION}
          name: classifier
    graph:
      name: classifier
      type: MODEL
    name: model
    replicas: 1
[26]:
!kubectl apply -f resources/model_seldon.yaml
seldondeployment.machinelearning.seldon.io/example-seldon configured
[27]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=example-seldon -o jsonpath='{.items[0].metadata.name}')
Waiting for deployment "example-seldon-model-0-classifier" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "example-seldon-model-0-classifier" rollout to finish: 1 old replicas are pending termination...
deployment "example-seldon-model-0-classifier" successfully rolled out
[28]:
for i in range(60):
    state=!kubectl get sdep example-seldon -o jsonpath='{.status.state}'
    state=state[0]
    print(state)
    if state=="Available":
        break
    time.sleep(1)
assert(state=="Available")
Available
[29]:
X=!curl -s -d '{"data": {"ndarray":[[1.0, 2.0, 5.0]]}}' \
   -X POST http://localhost:8003/seldon/seldon/example-seldon/api/v1.0/predictions \
   -H "Content-Type: application/json"
d=json.loads(X[0])
print(d)
assert(d["data"]["ndarray"][0][0] > 0.4)
{'data': {'names': ['proba'], 'ndarray': [[0.43782349911420193]]}, 'meta': {}}
[30]:
X=!cd ../executor/proto && grpcurl -d '{"data":{"ndarray":[[1.0,2.0,5.0]]}}' \
         -rpc-header seldon:example-seldon -rpc-header namespace:seldon \
         -plaintext \
         -proto ./prediction.proto  0.0.0.0:8003 seldon.protos.Seldon/Predict
d=json.loads("".join(X))
print(d)
assert(d["data"]["ndarray"][0][0] > 0.4)
{'meta': {}, 'data': {'names': ['proba'], 'ndarray': [[0.43782349911420193]]}}
[31]:
!kubectl delete -f resources/model_seldon.yaml
seldondeployment.machinelearning.seldon.io "example-seldon" deleted
[32]:
!kubectl delete -f ../servers/sklearnserver/samples/iris.yaml
seldondeployment.machinelearning.seldon.io "sklearn" deleted
[ ]: