This page was generated from examples/feedback/metrics-server/README.ipynb.

Stateful Model Feedback Metrics Server

In this example we will add statistical performance metrics capabilities by levering the Seldon metrics server.

Dependencies * Seldon Core installed * Ingress provider (Istio or Ambassador)

An easy way is to run examples/centralized-logging/full-kind-setup.sh and then:

helm delete seldon-core-loadtesting
helm delete seldon-single-model

Then port-forward to that ingress on localhost:8003 in a separate terminal either with:

Ambassador:

kubectl port-forward $(kubectl get pods -n seldon -l app.kubernetes.io/name=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8003:8080

Istio:

kubectl port-forward -n istio-system svc/istio-ingressgateway 8003:80
[1]:
!kubectl create namespace seldon || echo "namespace already created"
Error from server (AlreadyExists): namespaces "seldon" already exists
namespace already created
[2]:
!kubectl config set-context $(kubectl config current-context) --namespace=seldon
Context "kind-ansible" modified.
[3]:
!mkdir -p config

Create a simple model

We create a multiclass classification model - iris classifier.

The iris classifier takes an input array, and returns the prediction of the 4 classes.

The prediction can be done as numeric or as a probability array.

[4]:
%%bash
kubectl apply -f - << END
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: multiclass-model
spec:
  predictors:
  - graph:
      children: []
      implementation: SKLEARN_SERVER
      modelUri: gs://seldon-models/sklearn/iris
      name: classifier
      logger:
        url: http://seldon-multiclass-model-metrics.seldon.svc.cluster.local:80/
        mode: all
    name: default
    replicas: 1
END
seldondeployment.machinelearning.seldon.io/multiclass-model unchanged
[5]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=multiclass-model -o jsonpath='{.items[0].metadata.name}')
deployment "multiclass-model-default-0-classifier" successfully rolled out

Send test request

[6]:
res=!curl -X POST "http://localhost:8003/seldon/seldon/multiclass-model/api/v1.0/predictions" \
        -H "Content-Type: application/json" -d '{"data": { "ndarray": [[1,2,3,4]]}, "meta": { "puid": "hello" }}'
print(res)
import json
j=json.loads(res[-1])
assert(len(j["data"]["ndarray"][0])==3)
['  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current', '                                 Dload  Upload   Total   Spent    Left  Speed', '', '  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0', '100   267  100   203  100    64   9227   2909 --:--:-- --:--:-- --:--:-- 12136', '{"data":{"names":["t:0","t:1","t:2"],"ndarray":[[0.0006985194531162841,0.003668039039435755,0.9956334415074478]]},"meta":{"puid":"hello","requestPath":{"classifier":"seldonio/sklearnserver:1.7.0-dev"}}}']

Metrics Server

You can create a kubernetes deployment of the metrics server with this:

[7]:
%%writefile config/multiclass-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: seldon-multiclass-model-metrics
  namespace: seldon
  labels:
    app: seldon-multiclass-model-metrics
spec:
  replicas: 1
  selector:
    matchLabels:
      app: seldon-multiclass-model-metrics
  template:
    metadata:
      labels:
        app: seldon-multiclass-model-metrics
    spec:
      securityContext:
          runAsUser: 8888
      containers:
      - name: user-container
        image: seldonio/alibi-detect-server:1.7.0-dev
        imagePullPolicy: IfNotPresent
        args:
        - --model_name
        - multiclassserver
        - --http_port
        - '8080'
        - --protocol
        - seldonfeedback.http
        - --storage_uri
        - "adserver.cm_models.multiclass_one_hot.MulticlassOneHot"
        - --reply_url
        - http://message-dumper.default
        - --event_type
        - io.seldon.serving.feedback.metrics
        - --event_source
        - io.seldon.serving.feedback
        - MetricsServer
        env:
        - name: "SELDON_DEPLOYMENT_ID"
          value: "multiclass-model"
        - name: "PREDICTIVE_UNIT_ID"
          value: "classifier"
        - name: "PREDICTIVE_UNIT_IMAGE"
          value: "alibi-detect-server:1.7.0-dev"
        - name: "PREDICTOR_ID"
          value: "default"
---
apiVersion: v1
kind: Service
metadata:
  name: seldon-multiclass-model-metrics
  namespace: seldon
  labels:
    app: seldon-multiclass-model-metrics
spec:
  selector:
    app: seldon-multiclass-model-metrics
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
Overwriting config/multiclass-deployment.yaml
[8]:
!kubectl apply -n seldon -f config/multiclass-deployment.yaml
deployment.apps/seldon-multiclass-model-metrics unchanged
service/seldon-multiclass-model-metrics unchanged
[9]:
!kubectl rollout status deploy/seldon-multiclass-model-metrics
deployment "seldon-multiclass-model-metrics" successfully rolled out
[10]:
import time
time.sleep(20)

Send feedback

[11]:
res=!curl -X POST "http://localhost:8003/seldon/seldon/multiclass-model/api/v1.0/feedback" \
        -H "Content-Type: application/json" \
        -d '{"response": {"data": {"ndarray": [[0.0006985194531162841,0.003668039039435755,0.9956334415074478]]}}, "truth":{"data": {"ndarray": [[0,0,1]]}}}'
print(res)
import json
j=json.loads(res[-1])
assert("data" in j)
['  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current', '                                 Dload  Upload   Total   Spent    Left  Speed', '', '  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0', '100   251  100   107  100   144    136    183 --:--:-- --:--:-- --:--:--   320', '100   251  100   107  100   144    136    183 --:--:-- --:--:-- --:--:--   319', '{"data":{"tensor":{"shape":[0]}},"meta":{"requestPath":{"classifier":"seldonio/sklearnserver:1.7.0-dev"}}}']
[12]:
import time
time.sleep(3)

Check that metrics are recorded

[13]:
res=!kubectl logs $(kubectl get pods -l app=seldon-multiclass-model-metrics \
                    -n seldon -o jsonpath='{.items[0].metadata.name}') | grep "PROCESSING Feedback Event"
print(res)
assert(len(res)>0)
['[I 210304 12:16:36 cm_model:103] PROCESSING Feedback Event.']

Cleanup

[14]:
!kubectl delete -n seldon -f config/multiclass-deployment.yaml
deployment.apps "seldon-multiclass-model-metrics" deleted
service "seldon-multiclass-model-metrics" deleted
[15]:
!kubectl delete sdep multiclass-model
seldondeployment.machinelearning.seldon.io "multiclass-model" deleted