Basic Examples with Different Protocols

Prerequisites

  • A kubernetes cluster with kubectl configured

  • curl

  • grpcurl

  • pygmentize

Setup Seldon Core

Use the setup notebook to Setup Cluster to setup Seldon Core with an ingress - either Ambassador or Istio.

Then port-forward to that ingress on localhost:8003 in a separate terminal either with:

  • Ambassador: kubectl port-forward $(kubectl get pods -n seldon -l app.kubernetes.io/name=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8003:8080

  • Istio: kubectl port-forward $(kubectl get pods -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}') -n istio-system 8003:80

Install Seldon Analytics

[1]:
!helm install seldon-core-analytics ../../../helm-charts/seldon-core-analytics -n seldon-system --wait
NAME: seldon-core-analytics
LAST DEPLOYED: Sun Nov  8 10:24:13 2020
NAMESPACE: seldon-system
STATUS: deployed
REVISION: 1
[2]:
!kubectl create namespace seldon
Error from server (AlreadyExists): namespaces "seldon" already exists
[3]:
!kubectl config set-context $(kubectl config current-context) --namespace=seldon
Context "kind-kind" modified.
[4]:
import json
import time

Custom Metrics

[5]:
!pygmentize model.yaml
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: seldon-model
spec:
  name: test-deployment
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: seldonio/model-with-metrics:0.2
          name: classifier
    graph:
      children: []
      name: classifier
      type: MODEL
    name: example
    replicas: 1
[6]:
!kubectl create -f model.yaml
seldondeployment.machinelearning.seldon.io/seldon-model created
[7]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=seldon-model \
                                 -o jsonpath='{.items[0].metadata.name}')
Waiting for deployment "seldon-model-example-0-classifier" rollout to finish: 0 of 1 updated replicas are available...
deployment "seldon-model-example-0-classifier" successfully rolled out
[8]:
responseRaw=!curl -s -d '{"data": {"ndarray":[[1.0, 2.0, 5.0]]}}'    -X POST http://localhost:8003/seldon/seldon/seldon-model/api/v1.0/predictions    -H "Content-Type: application/json"
[9]:
response = json.loads(responseRaw[0])
print(response)
assert len(response["meta"]["metrics"]) == 3
{'data': {'names': ['t:0', 't:1', 't:2'], 'ndarray': [[1.0, 2.0, 5.0]]}, 'meta': {'metrics': [{'key': 'mycounter', 'type': 'COUNTER', 'value': 1}, {'key': 'mygauge', 'type': 'GAUGE', 'value': 100}, {'key': 'mytimer', 'type': 'TIMER', 'value': 20.2}]}}
[10]:
print("Waiting so metrics can be scraped")
time.sleep(30)
Waiting so metrics can be scraped
[11]:
%%writefile get-metrics.sh

kubectl run --quiet=true -it --rm curlmetrics --image=radial/busyboxplus:curl --restart=Never -- \
    curl -s seldon-core-analytics-prometheus-seldon.seldon-system/api/v1/query?query=mycounter_total
Overwriting get-metrics.sh
[12]:
responseRaw =! bash get-metrics.sh
[13]:
responseRaw[0]
[13]:
'{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"mycounter_total","app":"seldon-model-example-0-classifier","app_kubernetes_io_managed_by":"seldon-core","deployment_name":"seldon-model","fluentd":"true","image_name":"seldonio/model-with-metrics","image_version":"0.2","instance":"10.244.1.109:6000","job":"kubernetes-pods","kubernetes_namespace":"seldon","kubernetes_pod_name":"seldon-model-example-0-classifier-869f44f779-547dj","method":"predict","model_image":"seldonio/model-with-metrics","model_name":"classifier","model_version":"0.2","pod_template_hash":"869f44f779","predictor_name":"example","predictor_version":"example","seldon_app":"seldon-model-example","seldon_app_svc":"seldon-model-example-classifier","seldon_deployment_id":"seldon-model","seldon_deployment_name":"seldon-model","seldon_io_default":"true","seldon_io_model":"true","version":"example","worker_id":"40"},"value":[1604831146.35,"1"]}]}}'
[14]:
response = json.loads(responseRaw[0])
print(response)
assert response["data"]["result"][0]["metric"]["__name__"] == "mycounter_total"
{'status': 'success', 'data': {'resultType': 'vector', 'result': [{'metric': {'__name__': 'mycounter_total', 'app': 'seldon-model-example-0-classifier', 'app_kubernetes_io_managed_by': 'seldon-core', 'deployment_name': 'seldon-model', 'fluentd': 'true', 'image_name': 'seldonio/model-with-metrics', 'image_version': '0.2', 'instance': '10.244.1.109:6000', 'job': 'kubernetes-pods', 'kubernetes_namespace': 'seldon', 'kubernetes_pod_name': 'seldon-model-example-0-classifier-869f44f779-547dj', 'method': 'predict', 'model_image': 'seldonio/model-with-metrics', 'model_name': 'classifier', 'model_version': '0.2', 'pod_template_hash': '869f44f779', 'predictor_name': 'example', 'predictor_version': 'example', 'seldon_app': 'seldon-model-example', 'seldon_app_svc': 'seldon-model-example-classifier', 'seldon_deployment_id': 'seldon-model', 'seldon_deployment_name': 'seldon-model', 'seldon_io_default': 'true', 'seldon_io_model': 'true', 'version': 'example', 'worker_id': '40'}, 'value': [1604831146.35, '1']}]}}
[15]:
responseRaw=!cd ../../../executor/proto && grpcurl -d '{"data":{"ndarray":[[1.0,2.0,5.0]]}}' \
         -rpc-header seldon:seldon-model -rpc-header namespace:seldon \
         -plaintext \
         -proto ./prediction.proto  0.0.0.0:8003 seldon.protos.Seldon/Predict
[16]:
response = json.loads("".join(responseRaw))
print(response)
assert len(response["meta"]["metrics"]) == 3
{'meta': {'metrics': [{'key': 'mycounter', 'value': 1}, {'key': 'mygauge', 'type': 'GAUGE', 'value': 100}, {'key': 'mytimer', 'type': 'TIMER', 'value': 20.2}]}, 'data': {'names': ['t:0', 't:1', 't:2'], 'ndarray': [[1, 2, 5]]}}
[17]:
print("Waiting so metrics can be scraped")
time.sleep(30)
Waiting so metrics can be scraped
[18]:
responseRaw =! bash get-metrics.sh
[19]:
response = json.loads(responseRaw[0])
print(response)
assert response["data"]["result"][0]["metric"]["__name__"] == "mycounter_total"
assert (
    response["data"]["result"][0]["metric"]["image_name"]
    == "seldonio/model-with-metrics"
)
{'status': 'success', 'data': {'resultType': 'vector', 'result': [{'metric': {'__name__': 'mycounter_total', 'app': 'seldon-model-example-0-classifier', 'app_kubernetes_io_managed_by': 'seldon-core', 'deployment_name': 'seldon-model', 'fluentd': 'true', 'image_name': 'seldonio/model-with-metrics', 'image_version': '0.2', 'instance': '10.244.1.73:6000', 'job': 'kubernetes-pods', 'kubernetes_namespace': 'seldon', 'kubernetes_pod_name': 'seldon-model-example-0-classifier-869f44f779-hk8p2', 'method': 'predict', 'model_image': 'seldonio/model-with-metrics', 'model_name': 'classifier', 'model_version': '0.2', 'pod_template_hash': '869f44f779', 'predictor_name': 'example', 'predictor_version': 'example', 'seldon_app': 'seldon-model-example', 'seldon_app_svc': 'seldon-model-example-classifier', 'seldon_deployment_id': 'seldon-model', 'seldon_deployment_name': 'seldon-model', 'seldon_io_default': 'true', 'seldon_io_model': 'true', 'version': 'example', 'worker_id': '42'}, 'value': [1604666136.309, '1']}, {'metric': {'__name__': 'mycounter_total', 'app': 'seldon-model-example-0-classifier', 'app_kubernetes_io_managed_by': 'seldon-core', 'deployment_name': 'seldon-model', 'fluentd': 'true', 'image_name': 'seldonio/model-with-metrics', 'image_version': '0.2', 'instance': '10.244.1.73:6000', 'job': 'kubernetes-pods', 'kubernetes_namespace': 'seldon', 'kubernetes_pod_name': 'seldon-model-example-0-classifier-869f44f779-hk8p2', 'method': 'predict', 'model_image': 'seldonio/model-with-metrics', 'model_name': 'classifier', 'model_version': '0.2', 'pod_template_hash': '869f44f779', 'predictor_name': 'example', 'predictor_version': 'example', 'seldon_app': 'seldon-model-example', 'seldon_app_svc': 'seldon-model-example-classifier', 'seldon_deployment_id': 'seldon-model', 'seldon_deployment_name': 'seldon-model', 'seldon_io_default': 'true', 'seldon_io_model': 'true', 'version': 'example', 'worker_id': '31'}, 'value': [1604666136.309, '1']}]}}
[20]:
!kubectl delete -f model.yaml
seldondeployment.machinelearning.seldon.io "seldon-model" deleted
[21]:
!helm delete seldon-core-analytics --namespace seldon-system
release "seldon-core-analytics" uninstalled
[ ]: