This page was generated from examples/models/template_model_with_metrics/modelWithMetrics.ipynb.

Model with Metrics¶

Dependencies¶

pip install seldon-core

Summary of Custom Metrics¶

Example testing a model with custom metrics.

Metrics can be

A COUNTER : the returned value will increment the current value
A GAUGE : the returned value will overwrite the current value
A TIMER : a number of millisecs. Prometheus SUM and COUNT metrics will be created.

You need to provide a list of dictionaries each with the following:

a type : COUNTER, GAUGE, or TIMER
a key : a user defined key
a value : a float value

See example code below:

[1]:

!pygmentize ModelWithMetrics.py

class ModelWithMetrics(object):

    def __init__(self):
        print("Initialising")

    def predict(self,X,features_names):
        print("Predict called")
        return X

    def metrics(self):
        return [
            {"type":"COUNTER","key":"mycounter","value":1}, # a counter which will increase by the given value
            {"type":"GAUGE","key":"mygauge","value":100}, # a gauge which will be set to given value
            {"type":"TIMER","key":"mytimer","value":20.2}, # a timer which will add sum and count metrics - assumed millisecs
            ]

REST¶

[2]:

!s2i build -E environment_rest . seldonio/seldon-core-s2i-python3:0.13 model-with-metrics-rest:0.1

---> Installing application source...
Build completed successfully

[3]:

!docker run --name "model-with-metrics" -d --rm -p 5000:5000 model-with-metrics-rest:0.1

5c2a9f9337acefa041df77b1bf476fa69b46acb81dfae4a8bc0ffbbb12067be4

Test predict¶

[4]:

!seldon-core-tester contract.json 0.0.0.0 5000 -p

----------------------------------------
SENDING NEW REQUEST:

[[6.543 4.872 6.89  0.208]]
RECEIVED RESPONSE:
meta {
  metrics {
    key: "mycounter"
    value: 1.0
  }
  metrics {
    key: "mygauge"
    type: GAUGE
    value: 100.0
  }
  metrics {
    key: "mytimer"
    type: TIMER
    value: 20.200000762939453
  }
}
data {
  names: "t:0"
  names: "t:1"
  names: "t:2"
  names: "t:3"
  ndarray {
    values {
      list_value {
        values {
          number_value: 6.543
        }
        values {
          number_value: 4.872
        }
        values {
          number_value: 6.89
        }
        values {
          number_value: 0.208
        }
      }
    }
  }
}

[5]:

!docker rm model-with-metrics --force

model-with-metrics

gRPC¶

[6]:

!s2i build -E environment_grpc . seldonio/seldon-core-s2i-python3:0.13 model-with-metrics-grpc:0.1

---> Installing application source...
Build completed successfully

[7]:

!docker run --name "model-with-metrics" -d --rm -p 5000:5000 model-with-metrics-grpc:0.1

85ac344a48c56931ff3bef98ca02bb0d9f7e7198619fee933faf92d62c594080

Test predict¶

[8]:

!seldon-core-tester contract.json 0.0.0.0 5000 -p --grpc

----------------------------------------
SENDING NEW REQUEST:

[[7.607 2.626 3.027 2.034]]
RECEIVED RESPONSE:
meta {
  metrics {
    key: "mycounter"
    value: 1.0
  }
  metrics {
    key: "mygauge"
    type: GAUGE
    value: 100.0
  }
  metrics {
    key: "mytimer"
    type: TIMER
    value: 20.200000762939453
  }
}
data {
  names: "t:0"
  names: "t:1"
  names: "t:2"
  names: "t:3"
  ndarray {
    values {
      list_value {
        values {
          number_value: 7.607
        }
        values {
          number_value: 2.626
        }
        values {
          number_value: 3.027
        }
        values {
          number_value: 2.034
        }
      }
    }
  }
}

[9]:

!docker rm model-with-metrics --force

model-with-metrics

Test using Minikube¶

Due to a `minikube/s2i issue <https://github.com/SeldonIO/seldon-core/issues/253>`__ you will need `s2i >= 1.1.13 <https://github.com/openshift/source-to-image/releases/tag/v1.1.13>`__

[10]:

!minikube start --memory 4096

There is a newer version of minikube available (v0.35.0).  Download it here:
https://github.com/kubernetes/minikube/releases/tag/v0.35.0

To disable this notification, run the following:
minikube config set WantUpdateNotification false
😄  minikube v0.34.1 on linux (amd64)
🔥  Creating virtualbox VM (CPUs=2, Memory=4096MB, Disk=20000MB) ...
📶  "minikube" IP address is 192.168.99.100
🐳  Configuring Docker as the container runtime ...
✨  Preparing Kubernetes environment ...
🚜  Pulling images required by Kubernetes v1.13.3 ...
🚀  Launching Kubernetes v1.13.3 using kubeadm ...
🔑  Configuring cluster permissions ...
🤔  Verifying component health .....
💗  kubectl is now configured to use "minikube"
🏄  Done! Thank you for using minikube!

[11]:

!kubectl create clusterrolebinding kube-system-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default

clusterrolebinding.rbac.authorization.k8s.io/kube-system-cluster-admin created

[12]:

!helm init

$HELM_HOME has been configured at /home/clive/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!

[13]:

!kubectl rollout status deploy/tiller-deploy -n kube-system

Waiting for deployment "tiller-deploy" rollout to finish: 0 of 1 updated replicas are available...
deployment "tiller-deploy" successfully rolled out

[6]:

!helm install ../../../helm-charts/seldon-core-operator --name seldon-core --set usageMetrics.enabled=true   --namespace seldon-system

NAME:   seldon-core
LAST DEPLOYED: Tue Apr 16 13:52:58 2019
NAMESPACE: seldon-system
STATUS: DEPLOYED

RESOURCES:
==> v1/Service
NAME                                        TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)  AGE
seldon-operator-controller-manager-service  ClusterIP  10.100.187.38  <none>       443/TCP  0s

==> v1/ServiceAccount
NAME                        SECRETS  AGE
seldon-spartakus-volunteer  1        0s

==> v1beta1/ClusterRole
NAME                        AGE
seldon-spartakus-volunteer  0s

==> v1/Secret
NAME                                   TYPE    DATA  AGE
seldon-operator-webhook-server-secret  Opaque  0     0s

==> v1/ClusterRole
NAME                          AGE
seldon-operator-manager-role  0s

==> v1/ClusterRoleBinding
NAME                                 AGE
seldon-operator-manager-rolebinding  0s

==> v1beta1/Deployment
NAME                        DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
seldon-spartakus-volunteer  1        0        0           0          0s

==> v1/StatefulSet
NAME                                DESIRED  CURRENT  AGE
seldon-operator-controller-manager  1        1        0s

==> v1beta1/ClusterRoleBinding
NAME                        AGE
seldon-spartakus-volunteer  0s

==> v1/Pod(related)
NAME                                  READY  STATUS             RESTARTS  AGE
seldon-operator-controller-manager-0  0/1    ContainerCreating  0         0s

==> v1/ConfigMap
NAME                     DATA  AGE
seldon-spartakus-config  3     0s

==> v1beta1/CustomResourceDefinition
NAME                                         AGE
seldondeployments.machinelearning.seldon.io  0s


NOTES:
NOTES: TODO

[7]:

!helm install seldon-core-analytics --name seldon-core-analytics --set grafana_prom_admin_password=password --set persistence.enabled=false --repo https://storage.googleapis.com/seldon-charts

NAME:   seldon-core-analytics
LAST DEPLOYED: Tue Apr 16 13:53:06 2019
NAMESPACE: seldon
STATUS: DEPLOYED

RESOURCES:
==> v1/Service
NAME                      TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)       AGE
alertmanager              ClusterIP  10.107.187.67  <none>       80/TCP        0s
grafana-prom              NodePort   10.96.185.3    <none>       80:31405/TCP  0s
prometheus-node-exporter  ClusterIP  None           <none>       9100/TCP      0s
prometheus-seldon         ClusterIP  10.109.47.248  <none>       80/TCP        0s

==> v1beta1/DaemonSet
NAME                      DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR  AGE
prometheus-node-exporter  1        1        0      1           0          <none>         0s

==> v1/Secret
NAME                 TYPE    DATA  AGE
grafana-prom-secret  Opaque  1     1s

==> v1/ServiceAccount
NAME        SECRETS  AGE
prometheus  1        0s

==> v1/Job
NAME                            DESIRED  SUCCESSFUL  AGE
grafana-prom-import-dashboards  1        0           0s

==> v1beta1/Deployment
NAME                     DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
alertmanager-deployment  1        1        1           0          0s
grafana-prom-deployment  1        1        1           0          0s
prometheus-deployment    1        1        1           0          0s

==> v1/Pod(related)
NAME                                      READY  STATUS             RESTARTS  AGE
grafana-prom-import-dashboards-k5zmm      0/1    ContainerCreating  0         0s
alertmanager-deployment-7cd568f668-9fvhm  0/1    ContainerCreating  0         0s
grafana-prom-deployment-899b4dd7b-n664s   0/1    ContainerCreating  0         0s
prometheus-node-exporter-wzntd            0/1    Pending            0         0s
prometheus-deployment-7554c97586-5dfl7    0/1    Pending            0         0s

==> v1/ConfigMap
NAME                       DATA  AGE
alertmanager-server-conf   1     1s
grafana-import-dashboards  11    1s
prometheus-rules           0     0s
prometheus-server-conf     1     0s

==> v1beta1/ClusterRole
NAME        AGE
prometheus  0s

==> v1beta1/ClusterRoleBinding
NAME        AGE
prometheus  0s


NOTES:
NOTES: TODO

[8]:

!kubectl rollout status deploy/seldon-controller-manager -n seldon-system

partitioned roll out complete: 1 new pods have been updated...

Setup Ingress¶

Please note: There are reported gRPC issues with ambassador (see https://github.com/SeldonIO/seldon-core/issues/473).

[9]:

!helm install stable/ambassador --name ambassador --set crds.keep=false

NAME:   ambassador
LAST DEPLOYED: Tue Apr 16 13:53:59 2019
NAMESPACE: seldon
STATUS: DEPLOYED

RESOURCES:
==> v1beta1/ClusterRole
NAME        AGE
ambassador  0s

==> v1beta1/ClusterRoleBinding
NAME        AGE
ambassador  0s

==> v1/Service
NAME               TYPE          CLUSTER-IP      EXTERNAL-IP  PORT(S)                     AGE
ambassador-admins  ClusterIP     10.104.133.196  <none>       8877/TCP                    0s
ambassador         LoadBalancer  10.103.248.191  <pending>    80:31100/TCP,443:31733/TCP  0s

==> v1/Deployment
NAME        DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
ambassador  3        3        3           0          0s

==> v1/Pod(related)
NAME                         READY  STATUS             RESTARTS  AGE
ambassador-5b89d44544-25vlr  0/1    ContainerCreating  0         0s
ambassador-5b89d44544-dtnm6  0/1    ContainerCreating  0         0s
ambassador-5b89d44544-j7km2  0/1    ContainerCreating  0         0s

==> v1/ServiceAccount
NAME        SECRETS  AGE
ambassador  1        0s


NOTES:
Congratuations! You've successfully installed Ambassador.

For help, visit our Slack at https://d6e.co/slack or view the documentation online at https://www.getambassador.io.

To get the IP address of Ambassador, run the following commands:
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
     You can watch the status of by running 'kubectl get svc -w  --namespace seldon ambassador'

  On GKE/Azure:
  export SERVICE_IP=$(kubectl get svc --namespace seldon ambassador -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

  On AWS:
  export SERVICE_IP=$(kubectl get svc --namespace seldon ambassador -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

  echo http://$SERVICE_IP:

[10]:

!kubectl rollout status deployment.apps/ambassador

Waiting for deployment "ambassador" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "ambassador" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "ambassador" rollout to finish: 2 of 3 updated replicas are available...
deployment "ambassador" successfully rolled out

Port forward the dashboard when running kubectl port-forward $(kubectl get pods -n default -l app=grafana-prom-server -o jsonpath='{.items[0].metadata.name}') -n default 3000:3000
Visit http://localhost:3000/dashboard/db/prediction-analytics?refresh=5s&orgId=1 and login using “admin” and the password you set above when launching with helm.

REST¶

[11]:

!eval $(minikube docker-env) && s2i build -E environment_rest . seldonio/seldon-core-s2i-python3:0.13 model-with-metrics-rest:0.1

---> Installing application source...
Build completed successfully

[12]:

!kubectl create -f deployment-rest.json

seldondeployment.machinelearning.seldon.io/mymodel created

[13]:

!kubectl rollout status deploy/mymodel-mymodel-b79af31

Waiting for deployment "mymodel-mymodel-b79af31" rollout to finish: 0 of 1 updated replicas are available...
deployment "mymodel-mymodel-b79af31" successfully rolled out

Test predict¶

[14]:

!seldon-core-api-tester contract.json `minikube ip` `kubectl get svc ambassador -o jsonpath='{.spec.ports[0].nodePort}'` \
    mymodel --namespace seldon -p

----------------------------------------
SENDING NEW REQUEST:

[[7.564 2.469 4.288 1.109]]
RECEIVED RESPONSE:
meta {
  puid: "mqi1cen7kiv8nlqs8pj443d9ns"
  requestPath {
    key: "complex-model"
    value: "model-with-metrics-rest:0.1"
  }
  metrics {
    key: "mycounter"
    value: 1.0
  }
  metrics {
    key: "mygauge"
    type: GAUGE
    value: 100.0
  }
  metrics {
    key: "mytimer"
    type: TIMER
    value: 20.200000762939453
  }
}
data {
  names: "t:0"
  names: "t:1"
  names: "t:2"
  names: "t:3"
  ndarray {
    values {
      list_value {
        values {
          number_value: 7.564
        }
        values {
          number_value: 2.469
        }
        values {
          number_value: 4.288
        }
        values {
          number_value: 1.109
        }
      }
    }
  }
}

[15]:

!kubectl delete -f deployment-rest.json

seldondeployment.machinelearning.seldon.io "mymodel" deleted

gRPC¶

[16]:

!eval $(minikube docker-env) && s2i build -E environment_grpc . seldonio/seldon-core-s2i-python3:0.13 model-with-metrics-grpc:0.1

---> Installing application source...
Build completed successfully

[17]:

!kubectl create -f deployment-grpc.json

seldondeployment.machinelearning.seldon.io/mymodel created

[18]:

!kubectl rollout status deploy/mymodel-mymodel-5818788

Waiting for deployment "mymodel-mymodel-5818788" rollout to finish: 0 of 1 updated replicas are available...
deployment "mymodel-mymodel-5818788" successfully rolled out

Validate on Grafana¶

To check the metrics have appeared on Prometheus and are available in Grafana you could create a new graph in a dashboard and use the query:

mycounter_total

Test predict¶

[20]:

!seldon-core-api-tester contract.json `minikube ip` `kubectl get svc ambassador -o jsonpath='{.spec.ports[0].nodePort}'` \
    mymodel --namespace seldon -p --grpc

----------------------------------------
SENDING NEW REQUEST:

[[5.625 2.569 7.636 0.647]]
RECEIVED RESPONSE:
meta {
  puid: "mu75mrgolbub9n24v11eflp7jr"
  requestPath {
    key: "complex-model"
    value: "model-with-metrics-grpc:0.1"
  }
  metrics {
    key: "mycounter"
    value: 1.0
  }
  metrics {
    key: "mygauge"
    type: GAUGE
    value: 100.0
  }
  metrics {
    key: "mytimer"
    type: TIMER
    value: 20.200000762939453
  }
}
data {
  names: "t:0"
  names: "t:1"
  names: "t:2"
  names: "t:3"
  ndarray {
    values {
      list_value {
        values {
          number_value: 5.625
        }
        values {
          number_value: 2.569
        }
        values {
          number_value: 7.636
        }
        values {
          number_value: 0.647
        }
      }
    }
  }
}

[21]:

!kubectl delete -f deployment-grpc.json

seldondeployment.machinelearning.seldon.io "mymodel" deleted

[27]:

!minikube delete

🔥  Deleting "minikube" from virtualbox ...
💔  The "minikube" cluster has been deleted.

[ ]: