This page was generated from notebooks/timeouts.ipynb.

Testing Custom Timeouts

Setup Seldon Core

Use the setup notebook to Setup Cluster with Ambassador Ingress and Install Seldon Core. Instructions also online.

Illustration of both REST and gRPC requests.

[3]:
!pygmentize resources/model_short_timeouts.json
{
    "apiVersion": "machinelearning.seldon.io/v1alpha2",
    "kind": "SeldonDeployment",
    "metadata": {
        "labels": {
            "app": "seldon"
        },
        "name": "seldon-deployment-example"
    },
    "spec": {
        "annotations": {
            "project_name": "FX Market Prediction",
            "deployment_version": "v1",
            "seldon.io/rest-read-timeout":"1"
        },
        "name": "test-deployment",
        "oauth_key": "oauth-key",
        "oauth_secret": "oauth-secret",
        "predictors": [
            {
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/mock_classifier:1.0",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier",
                                "resources": {
                                    "requests": {
                                        "memory": "1Mi"
                                    }
                                }
                            }
                        ],
                        "terminationGracePeriodSeconds": 20
                    }
                }],
                "graph": {
                    "children": [],
                    "name": "classifier",
                    "endpoint": {
                        "type" : "REST"
                    },
                    "type": "MODEL"
                },
                "name": "fx-market-predictor",
                "replicas": 1,
                "annotations": {
                    "predictor_version" : "v1"
                }
            }
        ]
    }
}

Seldon Deployment with Short Timeout

Deploy the runtime graph to kubernetes.

[4]:
!kubectl apply -f resources/model_short_timeouts.json -n seldon
seldondeployment.machinelearning.seldon.io/seldon-deployment-example created
[5]:
!kubectl rollout status deploy/test-deployment-fx-market-predictor-7cd068f
Waiting for deployment "test-deployment-fx-market-predictor-7cd068f" rollout to finish: 0 of 1 updated replicas are available...
deployment "test-deployment-fx-market-predictor-7cd068f" successfully rolled out
[6]:
from seldon_core.seldon_client import SeldonClient
sc = SeldonClient(deployment_name="seldon-deployment-example",namespace="seldon")

This next request should fail as the timeout is too short

[7]:
r = sc.predict(gateway="ambassador",transport="rest")
print(r)
Success:False message:504:Gateway Timeout
Request:
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.24476905894230339
  }
}

Response:
None

Delete this graph and recreate one with a longer timeout

[8]:
!kubectl delete -f resources/model_short_timeouts.json
seldondeployment.machinelearning.seldon.io "seldon-deployment-example" deleted

Seldon Deployment with Long Timeout

[9]:
!pygmentize resources/model_long_timeouts.json
{
    "apiVersion": "machinelearning.seldon.io/v1alpha2",
    "kind": "SeldonDeployment",
    "metadata": {
        "labels": {
            "app": "seldon"
        },
        "name": "model-long-timeout"
    },
    "spec": {
        "annotations": {
            "deployment_version": "v1",
            "seldon.io/rest-read-timeout":"100000",
            "seldon.io/rest-connection-timeout":"100000",
            "seldon.io/grpc-read-timeout":"100000"
        },
        "name": "long-to",
        "oauth_key": "oauth-key",
        "oauth_secret": "oauth-secret",
        "predictors": [
            {
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/mock_classifier:1.0",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier",
                                "resources": {
                                    "requests": {
                                        "memory": "1Mi"
                                    }
                                }
                            }
                        ],
                        "terminationGracePeriodSeconds": 20
                    }
                }],
                "graph": {
                    "children": [],
                    "name": "classifier",
                    "endpoint": {
                        "type" : "REST"
                    },
                    "type": "MODEL"
                },
                "name": "test",
                "replicas": 1,
                "annotations": {
                    "predictor_version" : "v1"
                }
            }
        ]
    }
}
[10]:
!kubectl apply -f resources/model_long_timeouts.json -n seldon
seldondeployment.machinelearning.seldon.io/model-long-timeout created
[11]:
!kubectl rollout status deploy/long-to-test-7cd068f
Waiting for deployment "long-to-test-7cd068f" rollout to finish: 0 of 1 updated replicas are available...
deployment "long-to-test-7cd068f" successfully rolled out

This next request should work as the timeout is much longer

[12]:
r = sc.predict(gateway="ambassador",transport="rest",deployment_name="model-long-timeout")
print(r)
Success:True message:
Request:
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.5065588610378213
  }
}

Response:
meta {
  puid: "2murh7r9ud7rb3denhoa3f3ln2"
  requestPath {
    key: "classifier"
    value: "seldonio/mock_classifier:1.0"
  }
}
data {
  names: "proba"
  tensor {
    shape: 1
    shape: 1
    values: 0.08240516608630526
  }
}