Mock Classifier With Custom Endpoints Model

  • Wrap a basic python model for use as a prediction microservice in seldon-core
  • Run locally on Docker to test
  • Deploy on seldon-core running on minikube
  • Example of using custom endpoints that are scraped by prometheus

Dependencies

pip install seldon-core

Wrap model using s2i

Test locally using REST

[1]:
!make build_rest
s2i build -E environment_rest . seldonio/seldon-core-s2i-python3:0.4 seldonio/mock_classifier_with_custom_endpoints_rest:1.0
---> Installing application source...
---> Installing dependencies ...
Requirement already satisfied: numpy>=1.11.2 in /usr/local/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (1.15.4)
You are using pip version 18.1, however version 19.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Build completed successfully
[2]:
!docker run --name "mock_classifier_with_custom_endpoints_rest" -d --rm \
    -e PREDICTIVE_UNIT_SERVICE_PORT=5000 \
    -p 5000:5000 -p 5055:5055 seldonio/mock_classifier_with_custom_endpoints_rest:1.0
4c607cce90accb1595670f0d5353eccda6a9ab51dfc637ed7db988baac535ab0
[3]:
#
# Call the prediction endpoint.
# Send some random features that conform to the contract
#
!seldon-core-tester contract.json 0.0.0.0 5000 -p
----------------------------------------
SENDING NEW REQUEST:

[[-0.886 -1.141 -0.62 ]]
RECEIVED RESPONSE:
meta {
}
data {
  names: "proba"
  ndarray {
    values {
      list_value {
        values {
          number_value: 0.02190268365308545
        }
      }
    }
  }
}


[4]:
#
# Call the custom endpoint.
# In this example its used for the prediction call count.
#
!curl "http://localhost:5055/prometheus_metrics"
predict_call_count 1
[5]:
!docker rm -v "mock_classifier_with_custom_endpoints_rest" --force
mock_classifier_with_custom_endpoints_rest

Test locally using GRPC

[6]:
!make build_grpc
s2i build -E environment_grpc . seldonio/seldon-core-s2i-python3:0.4 seldonio/mock_classifier_with_custom_endpoints_grpc:1.0
---> Installing application source...
---> Installing dependencies ...
Requirement already satisfied: numpy>=1.11.2 in /usr/local/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (1.15.4)
You are using pip version 18.1, however version 19.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Build completed successfully
[7]:
!docker run --name "mock_classifier_with_custom_endpoints_rest" -d --rm \
    -e PREDICTIVE_UNIT_SERVICE_PORT=5000 \
    -p 5000:5000 -p 5055:5055 seldonio/mock_classifier_with_custom_endpoints_grpc:1.0
6fc646fb348a15a063f2230b854c82b5dc4d6921135d279881e3fd86460443f8
[8]:
#
# Call the prediction endpoint.
# Send some random features that conform to the contract,
# using NDArray.
#
!seldon-core-tester contract.json 0.0.0.0 5000 -p --grpc
----------------------------------------
SENDING NEW REQUEST:

[[-0.045  0.177  0.967]]
RECEIVED RESPONSE:
meta {
}
data {
  names: "proba"
  ndarray {
    values {
      list_value {
        values {
          number_value: 0.0724040949230513
        }
      }
    }
  }
}


[9]:
#
# Call the prediction endpoint.
# Send some random features that conform to the contract,
# using Tensor.
#
!seldon-core-tester contract.json 0.0.0.0 5000 -p --grpc --tensor
----------------------------------------
SENDING NEW REQUEST:

[[ 1.249 -0.125 -1.616]]
RECEIVED RESPONSE:
meta {
}
data {
  names: "proba"
  tensor {
    shape: 1
    shape: 1
    values: 0.043911817837546704
  }
}


[10]:
#
# Call the custom endpoint.
# In this example its used for the prediction call count.
#
!curl "http://localhost:5055/prometheus_metrics"
predict_call_count 2
[11]:
!docker rm -v "mock_classifier_with_custom_endpoints_rest" --force
mock_classifier_with_custom_endpoints_rest

Test using Minikube

Due to a `minikube/s2i issue <https://github.com/SeldonIO/seldon-core/issues/253>`__ you will need `s2i >= 1.1.13 <https://github.com/openshift/source-to-image/releases/tag/v1.1.13>`__

[ ]:
!minikube start --memory=8096
[12]:
#
# Create a cluster-admin role binding
#
!kubectl create clusterrolebinding kube-system-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
clusterrolebinding.rbac.authorization.k8s.io/kube-system-cluster-admin created
[13]:
!helm init
$HELM_HOME has been configured at /home/clive/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!
[15]:
!kubectl rollout status deploy/tiller-deploy -n kube-system
deployment "tiller-deploy" successfully rolled out
[16]:
!helm install ../../../helm-charts/seldon-core-operator --name seldon-core --set usageMetrics.enabled=true   --namespace seldon-system
NAME:   seldon-core
LAST DEPLOYED: Thu May  9 16:16:47 2019
NAMESPACE: seldon-system
STATUS: DEPLOYED

RESOURCES:
==> v1/Secret
NAME                                   TYPE    DATA  AGE
seldon-operator-webhook-server-secret  Opaque  0     1s

==> v1beta1/CustomResourceDefinition
NAME                                         AGE
seldondeployments.machinelearning.seldon.io  1s

==> v1/ClusterRole
seldon-operator-manager-role  1s

==> v1/ClusterRoleBinding
NAME                                 AGE
seldon-operator-manager-rolebinding  1s

==> v1/Service
NAME                                        TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)  AGE
seldon-operator-controller-manager-service  ClusterIP  10.101.198.177  <none>       443/TCP  1s

==> v1/StatefulSet
NAME                                DESIRED  CURRENT  AGE
seldon-operator-controller-manager  1        1        1s

==> v1/Pod(related)
NAME                                  READY  STATUS             RESTARTS  AGE
seldon-operator-controller-manager-0  0/1    ContainerCreating  0         1s


NOTES:
NOTES: TODO


[17]:
!kubectl rollout status deploy/seldon-controller-manager -n seldon-system
Waiting for 1 pods to be ready...
partitioned roll out complete: 1 new pods have been updated...
[18]:
!helm install ../../../helm-charts/seldon-core-analytics --name seldon-core-analytics \
    --set grafana_prom_admin_password=password \
    --set persistence.enabled=false \
    --set prometheus.service_type=NodePort
NAME:   seldon-core-analytics
LAST DEPLOYED: Thu May  9 16:17:01 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME                       DATA  AGE
alertmanager-server-conf   1     0s
grafana-import-dashboards  11    0s
prometheus-rules           0     0s
prometheus-server-conf     1     0s

==> v1beta1/ClusterRoleBinding
NAME        AGE
prometheus  0s

==> v1beta1/Deployment
NAME                     DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
alertmanager-deployment  1        1        1           0          0s
grafana-prom-deployment  1        1        1           0          0s
prometheus-deployment    1        1        1           0          0s

==> v1/Service
NAME                      TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)       AGE
alertmanager              ClusterIP  10.99.122.106   <none>       80/TCP        0s
grafana-prom              NodePort   10.96.98.54     <none>       80:31488/TCP  0s
prometheus-node-exporter  ClusterIP  None            <none>       9100/TCP      0s
prometheus-seldon         NodePort   10.110.230.107  <none>       80:30421/TCP  0s

==> v1beta1/DaemonSet
NAME                      DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR  AGE
prometheus-node-exporter  1        1        0      1           0          <none>         0s

==> v1/Secret
NAME                 TYPE    DATA  AGE
grafana-prom-secret  Opaque  1     0s

==> v1beta1/ClusterRole
NAME        AGE
prometheus  0s

==> v1/Job
NAME                            DESIRED  SUCCESSFUL  AGE
grafana-prom-import-dashboards  1        0           0s

==> v1/Pod(related)
NAME                                      READY  STATUS             RESTARTS  AGE
grafana-prom-import-dashboards-p95nt      0/1    ContainerCreating  0         0s
alertmanager-deployment-746bd758f9-dfcnm  0/1    ContainerCreating  0         0s
grafana-prom-deployment-899b4dd7b-h4rrs   0/1    ContainerCreating  0         0s
prometheus-node-exporter-kkfzr            0/1    Pending            0         0s
prometheus-deployment-659884c687-qm7rs    0/1    Pending            0         0s

==> v1/ServiceAccount
NAME        SECRETS  AGE
prometheus  1        0s


NOTES:
NOTES: TODO


Setup Ingress

Please note: There are reported gRPC issues with ambassador (see https://github.com/SeldonIO/seldon-core/issues/473).

[19]:
!helm install stable/ambassador --name ambassador --set crds.keep=false
NAME:   ambassador
LAST DEPLOYED: Thu May  9 16:17:06 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Service
NAME               TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)                     AGE
ambassador-admins  ClusterIP     10.97.163.140  <none>       8877/TCP                    0s
ambassador         LoadBalancer  10.107.81.103  <pending>    80:32071/TCP,443:31991/TCP  0s

==> v1/Deployment
NAME        DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
ambassador  3        3        3           0          0s

==> v1/Pod(related)
NAME                         READY  STATUS             RESTARTS  AGE
ambassador-5b89d44544-4sv7q  0/1    ContainerCreating  0         0s
ambassador-5b89d44544-9f87w  0/1    ContainerCreating  0         0s
ambassador-5b89d44544-djx76  0/1    ContainerCreating  0         0s

==> v1/ServiceAccount
NAME        SECRETS  AGE
ambassador  1        0s

==> v1beta1/ClusterRole
NAME        AGE
ambassador  0s

==> v1beta1/ClusterRoleBinding
NAME        AGE
ambassador  0s


NOTES:
Congratuations! You've successfully installed Ambassador.

For help, visit our Slack at https://d6e.co/slack or view the documentation online at https://www.getambassador.io.

To get the IP address of Ambassador, run the following commands:
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
     You can watch the status of by running 'kubectl get svc -w  --namespace default ambassador'

  On GKE/Azure:
  export SERVICE_IP=$(kubectl get svc --namespace default ambassador -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

  On AWS:
  export SERVICE_IP=$(kubectl get svc --namespace default ambassador -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

  echo http://$SERVICE_IP:

[20]:
!kubectl rollout status deployment.apps/ambassador
Waiting for deployment "ambassador" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "ambassador" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "ambassador" rollout to finish: 2 of 3 updated replicas are available...
deployment "ambassador" successfully rolled out
[21]:
!eval $(minikube docker-env) && make build_rest
s2i build -E environment_rest . seldonio/seldon-core-s2i-python3:0.4 seldonio/mock_classifier_with_custom_endpoints_rest:1.0
---> Installing application source...
---> Installing dependencies ...
Requirement already satisfied: numpy>=1.11.2 in /usr/local/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (1.15.4)
You are using pip version 18.1, however version 19.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Build completed successfully
[22]:
#
# Create moldel deployment
#
!kubectl create -f model-deployment.json
seldondeployment.machinelearning.seldon.io/seldon-deployment-example created
[23]:
!kubectl rollout status deploy/test-deployment-example-2582f05
deployment "test-deployment-example-2582f05" successfully rolled out
[33]:
!seldon-core-api-tester contract.json `minikube ip` `kubectl get svc ambassador -o jsonpath='{.spec.ports[0].nodePort}'` \
    seldon-deployment-example --namespace default -p
----------------------------------------
SENDING NEW REQUEST:

[[-0.01   1.065 -0.108]]
RECEIVED RESPONSE:
meta {
  puid: "mr94pflheljndmt4r7n9721qbf"
  requestPath {
    key: "classifier-1"
    value: "seldonio/mock_classifier_with_custom_endpoints_rest:1.0"
  }
}
data {
  names: "proba"
  ndarray {
    values {
      list_value {
        values {
          number_value: 0.06907408976710562
        }
      }
    }
  }
}


[34]:
#
# Query prometheus for the custom metrics
#
# A "custom_service" is run on port "5055" that
# is defined in model code "MeanClassifier.py".
#
# Prometeus set to scrape the model pod on port "5055"
# by the annotations in the "model-deployment.json" manifest.
#
!curl -s "$(minikube ip):$(kubectl get svc prometheus-seldon -o jsonpath='{.spec.ports[0].nodePort}')/api/v1/query?query=predict_call_count"
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"predict_call_count","app":"test-deployment-example-2582f05","instance":"172.17.0.17:5055","job":"kubernetes-pods","kubernetes_namespace":"default","kubernetes_pod_name":"test-deployment-example-2582f05-649c6c8794-xrnvq","pod_template_hash":"649c6c8794","seldon_app_classifier_1":"seldon-85d6383dfe46b983b73c236c56599001","seldon_deployment_id":"test-deployment-seldon-deployment-example"},"value":[1557418879.402,"1"]}]}}
[30]:
#
# The prometheus metrics can also been seen in a browser
# using the url shown after running this cell.
#
# In the prometheus ui execute the expression "predict_call_count"
#
!echo "http://$(minikube ip):$(kubectl get svc prometheus-seldon -o jsonpath='{.spec.ports[0].nodePort}')"
http://192.168.39.30:30421
[ ]:
#
# Clean up
#
!minikube delete
[ ]: