Basic Examples with Different Protocols

Prerequisites

  • A kubernetes cluster with kubectl configured

  • curl

  • grpcurl

  • pygmentize

Setup Seldon Core

Use the setup notebook to Setup Cluster to setup Seldon Core with an ingress - either Ambassador or Istio.

Then port-forward to that ingress on localhost:8003 in a separate terminal either with:

  • Ambassador: kubectl port-forward $(kubectl get pods -n seldon -l app.kubernetes.io/name=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8003:8080

  • Istio: kubectl port-forward $(kubectl get pods -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}') -n istio-system 8003:80

[1]:
!kubectl create namespace seldon
Error from server (AlreadyExists): namespaces "seldon" already exists
[2]:
!kubectl config set-context $(kubectl config current-context) --namespace=seldon
Context "gke-ansible" modified.
[3]:
import json
import time
[4]:
%%writefile get-metrics.sh

IP=$(kubectl get pods -l seldon-deployment-id=seldon-model -n seldon -o jsonpath='{.items[0].status.podIP}')
kubectl run --quiet=true -it --rm curlmetrics --image=radial/busyboxplus:curl --restart=Never -- \
    curl -s ${IP}:6000/prometheus | grep mycounter_total
Overwriting get-metrics.sh

Custom Metrics

[5]:
%%writefile model.yaml
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: seldon-model
spec:
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: seldonio/model-with-metrics:0.2
          name: classifier
    graph:
      children: []
      name: classifier
      type: MODEL
    name: example
    replicas: 1
Overwriting model.yaml
[6]:
!kubectl apply -f model.yaml
seldondeployment.machinelearning.seldon.io/seldon-model unchanged
[8]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=seldon-model \
                                 -o jsonpath='{.items[0].metadata.name}')
Waiting for deployment "seldon-model-example-0-classifier" rollout to finish: 0 of 1 updated replicas are available...
deployment "seldon-model-example-0-classifier" successfully rolled out

Send REST request

[9]:
responseRaw = !curl -s -H "Content-Type: application/json" -d '{"data": {"ndarray":[[1.0, 2.0, 5.0]]}}' http://localhost:8003/seldon/seldon/seldon-model/api/v1.0/predictions
[10]:
response = json.loads(responseRaw[0])
print(response)
assert len(response["meta"]["metrics"]) == 3
{'data': {'names': ['t:0', 't:1', 't:2'], 'ndarray': [[1.0, 2.0, 5.0]]}, 'meta': {'metrics': [{'key': 'mycounter', 'type': 'COUNTER', 'value': 1}, {'key': 'mygauge', 'type': 'GAUGE', 'value': 100}, {'key': 'mytimer', 'type': 'TIMER', 'value': 20.2}]}}
[11]:
responseRaw = ! bash get-metrics.sh
assert len(responseRaw) == 3

Send gRPC request

[12]:
responseRaw=!cd ../../../executor/proto && grpcurl -d '{"data":{"ndarray":[[1.0,2.0,5.0]]}}' \
         -rpc-header seldon:seldon-model -rpc-header namespace:seldon \
         -plaintext \
         -proto ./prediction.proto  0.0.0.0:8003 seldon.protos.Seldon/Predict
[13]:
response = json.loads("".join(responseRaw))
print(response)
assert len(response["meta"]["metrics"]) == 3
{'meta': {'metrics': [{'key': 'mycounter', 'value': 1}, {'key': 'mygauge', 'type': 'GAUGE', 'value': 100}, {'key': 'mytimer', 'type': 'TIMER', 'value': 20.2}]}, 'data': {'names': ['t:0', 't:1', 't:2'], 'ndarray': [[1, 2, 5]]}}
[15]:
responseRaw = ! bash get-metrics.sh
assert len(responseRaw) == 6
[16]:
!kubectl delete -f model.yaml
seldondeployment.machinelearning.seldon.io "seldon-model" deleted