Model with Metrics

Dependencies

pip install seldon-core

Summary of Custom Metrics

Example testing a model with custom metrics.

Metrics can be

  • A COUNTER : the returned value will increment the current value
  • A GAUGE : the returned value will overwrite the current value
  • A TIMER : a number of millisecs. Prometheus SUM and COUNT metrics will be created.

You need to provide a list of dictionaries each with the following:

  • a type : COUNTER, GAUGE, or TIMER
  • a key : a user defined key
  • a value : a float value

See example code below:

[1]:
!pygmentize ModelWithMetrics.py
class ModelWithMetrics(object):

    def __init__(self):
        print("Initialising")

    def predict(self, X, features_names):
        print("Predict called")
        return X

    def send_feedback(self, features, feature_names, reward, truth, routing=None):
        print("Send feedback called")
        return []

    def metrics(self):
        return [
            {"type": "COUNTER", "key": "mycounter", "value": 1}, # a counter which will increase by the given value
            {"type": "GAUGE", "key": "mygauge", "value": 100}, # a gauge which will be set to given value
            {"type": "TIMER", "key": "mytimer", "value": 20.2}, # a timer which will add sum and count metrics - assumed millisecs
        ]

REST

[2]:
!s2i build -E environment_rest . seldonio/seldon-core-s2i-python3:0.18 model-with-metrics-rest:0.1
---> Installing application source...
Build completed successfully
[3]:
!docker run --name "model-with-metrics" -d --rm -p 5000:5000 model-with-metrics-rest:0.1
3ee2d6a56c16ce323fd492b790b9d33117687a06816dc77b56c9a2a891b29614

Test predict

[4]:
!seldon-core-tester contract.json 0.0.0.0 5000 -p
----------------------------------------
SENDING NEW REQUEST:

[[4.59  4.024 8.461 0.493]]
RECEIVED RESPONSE:
meta {
  metrics {
    key: "mycounter"
    value: 1.0
  }
  metrics {
    key: "mygauge"
    type: GAUGE
    value: 100.0
  }
  metrics {
    key: "mytimer"
    type: TIMER
    value: 20.200000762939453
  }
}
data {
  names: "t:0"
  names: "t:1"
  names: "t:2"
  names: "t:3"
  ndarray {
    values {
      list_value {
        values {
          number_value: 4.59
        }
        values {
          number_value: 4.024
        }
        values {
          number_value: 8.461
        }
        values {
          number_value: 0.493
        }
      }
    }
  }
}


Test feedback

[5]:
!seldon-core-tester contract.json 0.0.0.0 5000 -p --endpoint send-feedback
----------------------------------------
SENDING NEW REQUEST:
RECEIVED RESPONSE:
Success:True message:
Request:
request {
  data {
    ndarray {
      values {
        list_value {
          values {
            number_value: 7.476
          }
          values {
            number_value: 2.005
          }
          values {
            number_value: 8.24
          }
          values {
            number_value: 2.771
          }
        }
      }
    }
  }
}
response {
  meta {
    metrics {
      key: "mycounter"
      value: 1.0
    }
    metrics {
      key: "mygauge"
      type: GAUGE
      value: 100.0
    }
    metrics {
      key: "mytimer"
      type: TIMER
      value: 20.200000762939453
    }
  }
  data {
    names: "t:0"
    names: "t:1"
    names: "t:2"
    names: "t:3"
    ndarray {
      values {
        list_value {
          values {
            number_value: 7.476
          }
          values {
            number_value: 2.005
          }
          values {
            number_value: 8.24
          }
          values {
            number_value: 2.771
          }
        }
      }
    }
  }
}
reward: 1.0

Response:
meta {
  metrics {
    key: "mycounter"
    value: 1.0
  }
  metrics {
    key: "mygauge"
    type: GAUGE
    value: 100.0
  }
  metrics {
    key: "mytimer"
    type: TIMER
    value: 20.200000762939453
  }
}
data {
  ndarray {
  }
}


Cleanup

[6]:
!docker rm model-with-metrics --force
model-with-metrics

gRPC

[7]:
!s2i build -E environment_grpc . seldonio/seldon-core-s2i-python3:0.18 model-with-metrics-grpc:0.1
---> Installing application source...
Build completed successfully
[8]:
!docker run --name "model-with-metrics" -d --rm -p 5000:5000 model-with-metrics-grpc:0.1
5c91a1febc69043ca77b43a39c23f282b77204a75fd1b7e4abbaea1f3638ac4e

Test predict

[9]:
!seldon-core-tester contract.json 0.0.0.0 5000 -p --grpc
----------------------------------------
SENDING NEW REQUEST:

[[4.426 3.448 8.844 2.65 ]]
RECEIVED RESPONSE:
meta {
  metrics {
    key: "mycounter"
    value: 1.0
  }
  metrics {
    key: "mygauge"
    type: GAUGE
    value: 100.0
  }
  metrics {
    key: "mytimer"
    type: TIMER
    value: 20.200000762939453
  }
}
data {
  names: "t:0"
  names: "t:1"
  names: "t:2"
  names: "t:3"
  ndarray {
    values {
      list_value {
        values {
          number_value: 4.426
        }
        values {
          number_value: 3.448
        }
        values {
          number_value: 8.844
        }
        values {
          number_value: 2.65
        }
      }
    }
  }
}


Test feedback

[10]:
!seldon-core-tester contract.json 0.0.0.0 5000 -p --endpoint send-feedback --grpc
----------------------------------------
SENDING NEW REQUEST:
RECEIVED RESPONSE:
Success:True message:
Request:
request {
  data {
    ndarray {
      values {
        list_value {
          values {
            number_value: 6.719
          }
          values {
            number_value: 3.262
          }
          values {
            number_value: 5.186
          }
          values {
            number_value: 0.305
          }
        }
      }
    }
  }
}
response {
  meta {
    metrics {
      key: "mycounter"
      value: 1.0
    }
    metrics {
      key: "mygauge"
      type: GAUGE
      value: 100.0
    }
    metrics {
      key: "mytimer"
      type: TIMER
      value: 20.200000762939453
    }
  }
  data {
    names: "t:0"
    names: "t:1"
    names: "t:2"
    names: "t:3"
    ndarray {
      values {
        list_value {
          values {
            number_value: 6.719
          }
          values {
            number_value: 3.262
          }
          values {
            number_value: 5.186
          }
          values {
            number_value: 0.305
          }
        }
      }
    }
  }
}
reward: 1.0

Response:
meta {
  metrics {
    key: "mycounter"
    value: 1.0
  }
  metrics {
    key: "mygauge"
    type: GAUGE
    value: 100.0
  }
  metrics {
    key: "mytimer"
    type: TIMER
    value: 20.200000762939453
  }
}
data {
  ndarray {
  }
}


Cleanup

[11]:
!docker rm model-with-metrics --force
model-with-metrics

Test using Minikube

Due to a `minikube/s2i issue <https://github.com/SeldonIO/seldon-core/issues/253>`__ you will need `s2i >= 1.1.13 <https://github.com/openshift/source-to-image/releases/tag/v1.1.13>`__

[ ]:
!minikube start --memory 4096

Setup Seldon Core

Use the setup notebook to Setup Cluster with Ambassador Ingress and Install Seldon Core. Instructions also online.

REST

[12]:
!eval $(minikube docker-env) && s2i build -E environment_rest . seldonio/seldon-core-s2i-python3:0.18 model-with-metrics-rest:0.1
---> Installing application source...
Build completed successfully
[13]:
!kubectl create -f deployment-rest.json
seldondeployment.machinelearning.seldon.io/mymodel created
[14]:
!kubectl rollout status deploy/mymodel-mymodel-b79af31
Waiting for deployment "mymodel-mymodel-b79af31" rollout to finish: 0 of 1 updated replicas are available...
deployment "mymodel-mymodel-b79af31" successfully rolled out

Test predict

[15]:
!seldon-core-api-tester contract.json `minikube ip` `kubectl get svc ambassador -n seldon -o jsonpath='{.spec.ports[0].nodePort}'` \
    mymodel --namespace seldon -p
----------------------------------------
SENDING NEW REQUEST:

[[5.727 4.073 9.725 2.436]]
RECEIVED RESPONSE:
meta {
  puid: "9ksv1ek9167r57h6ukgekasru"
  requestPath {
    key: "complex-model"
    value: "model-with-metrics-rest:0.1"
  }
  metrics {
    key: "mycounter"
    value: 1.0
  }
  metrics {
    key: "mygauge"
    type: GAUGE
    value: 100.0
  }
  metrics {
    key: "mytimer"
    type: TIMER
    value: 20.200000762939453
  }
}
data {
  names: "t:0"
  names: "t:1"
  names: "t:2"
  names: "t:3"
  ndarray {
    values {
      list_value {
        values {
          number_value: 5.727
        }
        values {
          number_value: 4.073
        }
        values {
          number_value: 9.725
        }
        values {
          number_value: 2.436
        }
      }
    }
  }
}


Test feedback

[17]:
!seldon-core-api-tester contract.json `minikube ip` `kubectl get svc ambassador -n seldon -o jsonpath='{.spec.ports[0].nodePort}'` \
    mymodel --namespace seldon -p --endpoint send-feedback
RECEIVED RESPONSE:
Success:True message:
Request:
request {
  meta {
  }
  data {
    tensor {
      shape: 1
      shape: 4
      values: 5.403
      values: 2.607
      values: 9.65
      values: 2.031
    }
  }
}
response {
  meta {
    puid: "m84pdl0kjfsf1mma71itarij7j"
    requestPath {
      key: "complex-model"
      value: "model-with-metrics-rest:0.1"
    }
    metrics {
      key: "mycounter"
      value: 1.0
    }
    metrics {
      key: "mygauge"
      type: GAUGE
      value: 100.0
    }
    metrics {
      key: "mytimer"
      type: TIMER
      value: 20.200000762939453
    }
  }
  data {
    names: "t:0"
    names: "t:1"
    names: "t:2"
    names: "t:3"
    tensor {
      shape: 1
      shape: 4
      values: 5.403
      values: 2.607
      values: 9.65
      values: 2.031
    }
  }
}
reward: 1.0

Response:


Cleanup

[18]:
!kubectl delete -f deployment-rest.json
seldondeployment.machinelearning.seldon.io "mymodel" deleted

gRPC

[19]:
!eval $(minikube docker-env) && s2i build -E environment_grpc . seldonio/seldon-core-s2i-python3:0.18 model-with-metrics-grpc:0.1
---> Installing application source...
Build completed successfully
[20]:
!kubectl create -f deployment-grpc.json
seldondeployment.machinelearning.seldon.io/mymodel created
[21]:
!kubectl rollout status deploy/mymodel-mymodel-5818788
Waiting for deployment "mymodel-mymodel-5818788" rollout to finish: 0 of 1 updated replicas are available...
deployment "mymodel-mymodel-5818788" successfully rolled out

Validate on Grafana

To check the metrics have appeared on Prometheus and are available in Grafana you could create a new graph in a dashboard and use the query:

mycounter_total

Test predict

[22]:
!seldon-core-api-tester contract.json `minikube ip` `kubectl get svc ambassador -n seldon -o jsonpath='{.spec.ports[0].nodePort}'` \
    mymodel --namespace seldon -p --grpc
----------------------------------------
SENDING NEW REQUEST:

[[6.143 2.272 1.225 1.684]]
RECEIVED RESPONSE:
meta {
  puid: "5f0bbii04bpn162duh3d59e7bu"
  requestPath {
    key: "complex-model"
    value: "model-with-metrics-grpc:0.1"
  }
  metrics {
    key: "mycounter"
    value: 1.0
  }
  metrics {
    key: "mygauge"
    type: GAUGE
    value: 100.0
  }
  metrics {
    key: "mytimer"
    type: TIMER
    value: 20.200000762939453
  }
}
data {
  names: "t:0"
  names: "t:1"
  names: "t:2"
  names: "t:3"
  ndarray {
    values {
      list_value {
        values {
          number_value: 6.143
        }
        values {
          number_value: 2.272
        }
        values {
          number_value: 1.225
        }
        values {
          number_value: 1.684
        }
      }
    }
  }
}


Test feedback

[23]:
!seldon-core-api-tester contract.json `minikube ip` `kubectl get svc ambassador -n seldon -o jsonpath='{.spec.ports[0].nodePort}'` \
    mymodel --namespace seldon -p --grpc --endpoint send-feedback
RECEIVED RESPONSE:
Success:True message:
Request:
request {
  meta {
  }
  data {
    tensor {
      shape: 1
      shape: 4
      values: 4.135
      values: 3.617
      values: 1.902
      values: 1.774
    }
  }
}
response {
  meta {
    puid: "gcav81kcoqg5nu41hkfpf8e37b"
    requestPath {
      key: "complex-model"
      value: "model-with-metrics-grpc:0.1"
    }
    metrics {
      key: "mycounter"
      value: 1.0
    }
    metrics {
      key: "mygauge"
      type: GAUGE
      value: 100.0
    }
    metrics {
      key: "mytimer"
      type: TIMER
      value: 20.200000762939453
    }
  }
  data {
    names: "t:0"
    names: "t:1"
    names: "t:2"
    names: "t:3"
    tensor {
      shape: 1
      shape: 4
      values: 4.135
      values: 3.617
      values: 1.902
      values: 1.774
    }
  }
}
reward: 1.0

Response:


Cleanup

[24]:
!kubectl delete -f deployment-grpc.json
seldondeployment.machinelearning.seldon.io "mymodel" deleted

Delete Minikube cluster

[25]:
!minikube delete
🔥  Deleting "minikube" from virtualbox ...
💔  The "minikube" cluster has been deleted.