This page was generated from notebooks/protocol_examples.ipynb.

Basic Examples with Different Protocols

Prerequisites

  • A kubernetes cluster with kubectl configured

  • curl

  • grpcurl

  • pygmentize

Examples

Setup Seldon Core

Use the setup notebook to Setup Cluster to setup Seldon Core with an ingress - either Ambassador or Istio.

Then port-forward to that ingress on localhost:8003 in a separate terminal either with:

  • Ambassador: kubectl port-forward $(kubectl get pods -n seldon -l app.kubernetes.io/name=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8003:8080

  • Istio: kubectl port-forward $(kubectl get pods -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}') -n istio-system 8003:80

[1]:
!kubectl create namespace seldon
Error from server (AlreadyExists): namespaces "seldon" already exists
[2]:
!kubectl config set-context $(kubectl config current-context) --namespace=seldon
Context "kind-kind" modified.
[3]:
import json
import time
[4]:
from IPython.core.magic import register_line_cell_magic

@register_line_cell_magic
def writetemplate(line, cell):
    with open(line, 'w') as f:
        f.write(cell.format(**globals()))
[5]:
VERSION=!cat ../version.txt
VERSION=VERSION[0]
VERSION
[5]:
'1.6.0-dev'

Seldon Protocol Model

We will deploy a REST model that uses the SELDON Protocol namely by specifying the attribute protocol: seldon

[6]:
%%writetemplate resources/model_seldon.yaml
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: example-seldon
spec:
  protocol: seldon
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: seldonio/mock_classifier:{VERSION}
          name: classifier
    graph:
      name: classifier
      type: MODEL
    name: model
    replicas: 1
[7]:
!kubectl apply -f resources/model_seldon.yaml
seldondeployment.machinelearning.seldon.io/example-seldon created
[8]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=example-seldon -o jsonpath='{.items[0].metadata.name}')
Waiting for deployment "example-seldon-model-0-classifier" rollout to finish: 0 of 1 updated replicas are available...
deployment "example-seldon-model-0-classifier" successfully rolled out
[9]:
for i in range(60):
    state=!kubectl get sdep example-seldon -o jsonpath='{.status.state}'
    state=state[0]
    print(state)
    if state=="Available":
        break
    time.sleep(1)
assert(state=="Available")
Available
[11]:
X=!curl -s -d '{"data": {"ndarray":[[1.0, 2.0, 5.0]]}}' \
   -X POST http://localhost:8003/seldon/seldon/example-seldon/api/v1.0/predictions \
   -H "Content-Type: application/json"
d=json.loads(X[0])
print(d)
assert(d["data"]["ndarray"][0][0] > 0.4)
{'data': {'names': ['proba'], 'ndarray': [[0.43782349911420193]]}, 'meta': {'requestPath': {'classifier': 'seldonio/mock_classifier:1.6.0-dev'}}}
[12]:
X=!cd ../executor/proto && grpcurl -d '{"data":{"ndarray":[[1.0,2.0,5.0]]}}' \
         -rpc-header seldon:example-seldon -rpc-header namespace:seldon \
         -plaintext \
         -proto ./prediction.proto  0.0.0.0:8003 seldon.protos.Seldon/Predict
d=json.loads("".join(X))
print(d)
assert(d["data"]["ndarray"][0][0] > 0.4)
{'meta': {'requestPath': {'classifier': 'seldonio/mock_classifier:1.6.0-dev'}}, 'data': {'names': ['proba'], 'ndarray': [[0.43782349911420193]]}}
[13]:
!kubectl delete -f resources/model_seldon.yaml
seldondeployment.machinelearning.seldon.io "example-seldon" deleted

Tensorflow Protocol Model

We will deploy a model that uses the TENSORLFOW Protocol namely by specifying the attribute protocol: tensorflow

[18]:
%%writefile resources/model_tfserving.yaml
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: example-tfserving
spec:
  protocol: tensorflow
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - args:
          - --port=8500
          - --rest_api_port=8501
          - --model_name=halfplustwo
          - --model_base_path=gs://seldon-models/tfserving/half_plus_two
          image: tensorflow/serving
          name: halfplustwo
          ports:
          - containerPort: 8501
            name: http
            protocol: TCP
          - containerPort: 8500
            name: grpc
            protocol: TCP
    graph:
      name: halfplustwo
      type: MODEL
      endpoint:
        httpPort: 8501
        grpcPort: 8500
    name: model
    replicas: 1
Overwriting resources/model_tfserving.yaml
[19]:
!kubectl apply -f resources/model_tfserving.yaml
seldondeployment.machinelearning.seldon.io/example-tfserving created
[20]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=example-tfserving \
                                 -o jsonpath='{.items[0].metadata.name}')
deployment "example-tfserving-model-0-halfplustwo" successfully rolled out
[21]:
for i in range(60):
    state=!kubectl get sdep example-tfserving -o jsonpath='{.status.state}'
    state=state[0]
    print(state)
    if state=="Available":
        break
    time.sleep(1)
assert(state=="Available")
Available
[22]:
X=!curl -s -d '{"instances": [1.0, 2.0, 5.0]}' \
   -X POST http://localhost:8003/seldon/seldon/example-tfserving/v1/models/halfplustwo/:predict \
   -H "Content-Type: application/json"
d=json.loads("".join(X))
print(d)
assert(d["predictions"][0] == 2.5)
{'predictions': [2.5, 3.0, 4.5]}
[23]:
X=!cd ../executor/proto && grpcurl \
   -d '{"model_spec":{"name":"halfplustwo"},"inputs":{"x":{"dtype": 1, "tensor_shape": {"dim":[{"size": 3}]}, "floatVal" : [1.0, 2.0, 3.0]}}}' \
   -rpc-header seldon:example-tfserving -rpc-header namespace:seldon \
   -plaintext -proto ./prediction_service.proto \
   0.0.0.0:8003 tensorflow.serving.PredictionService/Predict
d=json.loads("".join(X))
print(d)
assert(d["outputs"]["x"]["floatVal"][0] == 2.5)
{'outputs': {'x': {'dtype': 'DT_FLOAT', 'tensorShape': {'dim': [{'size': '3'}]}, 'floatVal': [2.5, 3, 3.5]}}, 'modelSpec': {'name': 'halfplustwo', 'version': '123', 'signatureName': 'serving_default'}}
[24]:
!kubectl delete -f resources/model_tfserving.yaml
seldondeployment.machinelearning.seldon.io "example-tfserving" deleted

KFServing V2 Protocol Model

We will deploy a REST model that uses the KFServing V2 Protocol namely by specifying the attribute protocol: kfserving

[25]:
%%writefile resources/model_v2.yaml
apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  name: triton
spec:
  protocol: kfserving
  predictors:
  - graph:
      children: []
      implementation: TRITON_SERVER
      modelUri: gs://seldon-models/trtis/simple-model
      name: simple
    name: simple
    replicas: 1
Overwriting resources/model_v2.yaml
[26]:
!kubectl apply -f resources/model_v2.yaml
seldondeployment.machinelearning.seldon.io/triton created
[27]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=triton -o jsonpath='{.items[0].metadata.name}')
Waiting for deployment "triton-simple-0-simple" rollout to finish: 0 of 1 updated replicas are available...
deployment "triton-simple-0-simple" successfully rolled out
[28]:
for i in range(60):
    state=!kubectl get sdep triton -o jsonpath='{.status.state}'
    state=state[0]
    print(state)
    if state=="Available":
        break
    time.sleep(1)
assert(state=="Available")
Available
[29]:
X=!curl -s -d '{"inputs":[{"name":"INPUT0","data":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16],"datatype":"INT32","shape":[1,16]},{"name":"INPUT1","data":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16],"datatype":"INT32","shape":[1,16]}]}'  \
        -X POST http://0.0.0.0:8003/seldon/seldon/triton/v2/models/simple/infer \
        -H "Content-Type: application/json"
d=json.loads(X[0])
print(d)
assert(d["outputs"][0]["data"][0]==2)
{'model_name': 'simple', 'model_version': '1', 'outputs': [{'name': 'OUTPUT0', 'datatype': 'INT32', 'shape': [1, 16], 'data': [2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32]}, {'name': 'OUTPUT1', 'datatype': 'INT32', 'shape': [1, 16], 'data': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}]}
[30]:
X=!cd ../executor/api/grpc/kfserving/inference && \
        grpcurl -d '{"model_name":"simple","inputs":[{"name":"INPUT0","contents":{"int_contents":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]},"datatype":"INT32","shape":[1,16]},{"name":"INPUT1","contents":{"int_contents":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]},"datatype":"INT32","shape":[1,16]}]}' \
        -plaintext -proto ./grpc_service.proto \
        -rpc-header seldon:triton -rpc-header namespace:seldon \
        0.0.0.0:8003 inference.GRPCInferenceService/ModelInfer
X="".join(X)
print(X)
{  "modelName": "simple",  "modelVersion": "1",  "outputs": [    {      "name": "OUTPUT0",      "datatype": "INT32",      "shape": [        "1",        "16"      ]    },    {      "name": "OUTPUT1",      "datatype": "INT32",      "shape": [        "1",        "16"      ]    }  ],  "rawOutputContents": [    "AgAAAAQAAAAGAAAACAAAAAoAAAAMAAAADgAAABAAAAASAAAAFAAAABYAAAAYAAAAGgAAABwAAAAeAAAAIAAAAA==",    "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="  ]}
[31]:
!kubectl delete -f resources/model_v2.yaml
seldondeployment.machinelearning.seldon.io "triton" deleted
[ ]: