This page was generated from notebooks/timeouts.ipynb.

Testing Custom Timeouts

Prerequisites

  • Kubernetes cluster with kubectl authorized

  • curl and grpcurl installed

Setup Seldon Core

Use the setup notebook to Setup Cluster to setup Seldon Core with an ingress - either Ambassador or Istio.

Then port-forward to that ingress on localhost:8003 in a separate terminal either with:

  • Ambassador: kubectl port-forward $(kubectl get pods -n seldon -l app.kubernetes.io/name=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8003:8080

  • Istio: kubectl port-forward $(kubectl get pods -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}') -n istio-system 8003:8080

[38]:
!kubectl create namespace seldon
namespace/seldon created
[39]:
!kubectl config set-context $(kubectl config current-context) --namespace=seldon
Context "kind-kind" modified.
[40]:
from IPython.core.magic import register_line_cell_magic

@register_line_cell_magic
def writetemplate(line, cell):
    with open(line, 'w') as f:
        f.write(cell.format(**globals()))
[41]:
VERSION=!cat ../version.txt
VERSION=VERSION[0]
VERSION
[41]:
'1.6.0-dev'

Configure Istio

For this example we will create the default istio gateway for seldon which needs to be called seldon-gateway. You can supply your own gateway by adding to your SeldonDeployments resources the annotation seldon.io/istio-gateway with values the name of your istio gateway.

[42]:
%%writefile resources/seldon-gateway.yaml
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: seldon-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
Overwriting resources/seldon-gateway.yaml
[43]:
!kubectl create -f resources/seldon-gateway.yaml -n istio-system
gateway.networking.istio.io/seldon-gateway created

Short timeouts deployment file

Below is the outputs of the Seldon config file required to set custom timeouts.

First we’ll show how we can set a short lived example by setting the "seldon.io/rest-read-timeout":"1" annotation.

[57]:
%%writetemplate resources/model_short_timeouts.json
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: seldon-model
spec:
  annotations:
    seldon.io/rest-timeout: "1000"
    seldon.io/grpc-timeout: "1000"
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: seldonio/mock_classifier:1.6.0-dev
          name: classifier
    graph:
      children: []
      name: classifier
      type: MODEL
      parameters:
      - name: delaySecs
        type: INT
        value: "2"
    name: example
    replicas: 1

We can then apply this Seldon Deployment file we defined above in the seldon namespace.

[58]:
!kubectl apply -f resources/model_short_timeouts.json -n seldon
seldondeployment.machinelearning.seldon.io/seldon-model configured

And wait until the resource runs correctly.

[59]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=seldon-model -o jsonpath='{.items[0].metadata.name}')
deployment "seldon-model-example-0-classifier" successfully rolled out
[60]:
!curl -i -d '{"data": {"ndarray":[[1.0, 2.0, 5.0]]}}'    -X POST http://localhost:8003/seldon/seldon/seldon-model/api/v1.0/predictions    -H "Content-Type: application/json"











[62]:
!cd ../executor/proto && grpcurl -d '{"data":{"ndarray":[[1.0,2.0,5.0]]}}' \
         -rpc-header seldon:seldon-model -rpc-header namespace:seldon \
         -plaintext \
         -proto ./prediction.proto  0.0.0.0:8003 seldon.protos.Seldon/Predict
ERROR:
  Code: DeadlineExceeded
  Message: Deadline Exceeded

Delete this graph and recreate one with a longer timeout

[63]:
!kubectl delete -f resources/model_short_timeouts.json
seldondeployment.machinelearning.seldon.io "seldon-model" deleted

Seldon Deployment with Long Timeout

Now we can have a look at how we can set a deployment with a longer timeout.

[64]:
%%writetemplate resources/model_long_timeouts.json
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: seldon-model
spec:
  annotations:
    seldon.io/rest-timeout: "5000"
    seldon.io/grpc-timeout: "5000"
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: seldonio/mock_classifier:1.6.0-dev
          name: classifier
    graph:
      children: []
      name: classifier
      type: MODEL
      parameters:
      - name: delaySecs
        type: INT
        value: "2"
    name: example
    replicas: 1

And we can apply it to deploy the model with long timeouts.

[65]:
!kubectl apply -f resources/model_long_timeouts.json -n seldon
seldondeployment.machinelearning.seldon.io/seldon-model created
[66]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=seldon-model -o jsonpath='{.items[0].metadata.name}')
Waiting for deployment "seldon-model-example-0-classifier" rollout to finish: 0 of 1 updated replicas are available...
deployment "seldon-model-example-0-classifier" successfully rolled out

This next request will work given that the timeout is much longer.

[67]:
!curl -i -d '{"data": {"ndarray":[[1.0, 2.0, 5.0]]}}'    -X POST http://localhost:8003/seldon/seldon/seldon-model/api/v1.0/predictions    -H "Content-Type: application/json"












{"data":{"names":["proba"],"ndarray":[[0.43782349911420193]]},"meta":{"requestPath":{"classifier":"seldonio/mock_classifier:1.6.0-dev"}}}
[68]:
!cd ../executor/proto && grpcurl -d '{"data":{"ndarray":[[1.0,2.0,5.0]]}}' \
         -rpc-header seldon:seldon-model -rpc-header namespace:seldon \
         -plaintext \
         -proto ./prediction.proto  0.0.0.0:8003 seldon.protos.Seldon/Predict
{
  "meta": {
    "requestPath": {
      "classifier": "seldonio/mock_classifier:1.6.0-dev"
    }
  },
  "data": {
    "names": [
      "proba"
    ],
    "ndarray": [
        [
              0.43782349911420193
            ]
      ]
  }
}
[69]:
!kubectl delete -f resources/model_long_timeouts.json
seldondeployment.machinelearning.seldon.io "seldon-model" deleted
[ ]: