This page was generated from notebooks/timeouts.ipynb.

Testing Custom Timeouts

Prerequistes

You will need

Create Cluster

Start minikube and ensure custom resource validation is activated and there is 5G of memory.

Your start command with the kvm driver would then look like:

minikube start --vm-driver kvm2 --memory 4096

Setup

[1]:
!kubectl create namespace seldon
namespace/seldon created
[2]:
!kubectl config set-context $(kubectl config current-context) --namespace=seldon
Context "minikube" modified.
[3]:
!kubectl create clusterrolebinding kube-system-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
clusterrolebinding.rbac.authorization.k8s.io/kube-system-cluster-admin created

Install Helm

[4]:
!kubectl -n kube-system create sa tiller
!kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
!helm init --service-account tiller
serviceaccount/tiller created
clusterrolebinding.rbac.authorization.k8s.io/tiller created
$HELM_HOME has been configured at /home/clive/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!
[5]:
!kubectl rollout status deploy/tiller-deploy -n kube-system
Waiting for deployment "tiller-deploy" rollout to finish: 0 of 1 updated replicas are available...
deployment "tiller-deploy" successfully rolled out

Start seldon-core

Install the custom resource definition

[2]:
!helm install ../helm-charts/seldon-core-operator --name seldon-core --set usageMetrics.enabled=true --namespace seldon-system
NAME:   seldon-core
LAST DEPLOYED: Tue Apr 16 08:08:43 2019
NAMESPACE: seldon-system
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME                     DATA  AGE
seldon-spartakus-config  3     0s

==> v1/ClusterRoleBinding
NAME                                 AGE
seldon-operator-manager-rolebinding  0s

==> v1beta1/Deployment
NAME                        DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
seldon-spartakus-volunteer  1        0        0           0          0s

==> v1/ServiceAccount
NAME                        SECRETS  AGE
seldon-spartakus-volunteer  1        0s

==> v1beta1/ClusterRoleBinding
NAME                        AGE
seldon-spartakus-volunteer  0s

==> v1/Pod(related)
NAME                                  READY  STATUS             RESTARTS  AGE
seldon-operator-controller-manager-0  0/1    ContainerCreating  0         0s

==> v1/Secret
NAME                                   TYPE    DATA  AGE
seldon-operator-webhook-server-secret  Opaque  0     0s

==> v1beta1/CustomResourceDefinition
NAME                                         AGE
seldondeployments.machinelearning.seldon.io  0s

==> v1/ClusterRole
seldon-operator-manager-role  0s

==> v1/Service
NAME                                        TYPE       CLUSTER-IP    EXTERNAL-IP  PORT(S)  AGE
seldon-operator-controller-manager-service  ClusterIP  10.97.51.246  <none>       443/TCP  0s

==> v1/StatefulSet
NAME                                DESIRED  CURRENT  AGE
seldon-operator-controller-manager  1        1        0s

==> v1beta1/ClusterRole
NAME                        AGE
seldon-spartakus-volunteer  0s


NOTES:
NOTES: TODO


Check all services are running before proceeding.

[3]:
!kubectl rollout status statefulset.apps/seldon-operator-controller-manager -n seldon-system
partitioned roll out complete: 1 new pods have been updated...

Setup Ingress

There are gRPC issues with the latest Ambassador, so we rewcommend 0.40.2 until these are fixed.

[5]:
!helm install stable/ambassador --name ambassador --set image.tag=0.40.2
NAME:   ambassador
LAST DEPLOYED: Tue Apr 16 08:09:32 2019
NAMESPACE: seldon
STATUS: DEPLOYED

RESOURCES:
==> v1/Service
NAME               TYPE          CLUSTER-IP    EXTERNAL-IP  PORT(S)                     AGE
ambassador-admins  ClusterIP     10.99.21.17   <none>       8877/TCP                    0s
ambassador         LoadBalancer  10.96.47.162  <pending>    80:30647/TCP,443:32371/TCP  0s

==> v1/Deployment
NAME        DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
ambassador  3        3        3           0          0s

==> v1/Pod(related)
NAME                         READY  STATUS             RESTARTS  AGE
ambassador-5b89d44544-9wbj6  0/1    ContainerCreating  0         0s
ambassador-5b89d44544-cd2t5  0/1    ContainerCreating  0         0s
ambassador-5b89d44544-rscdp  0/1    ContainerCreating  0         0s

==> v1/ServiceAccount
NAME        SECRETS  AGE
ambassador  1        0s

==> v1beta1/ClusterRole
NAME        AGE
ambassador  0s

==> v1beta1/ClusterRoleBinding
NAME        AGE
ambassador  0s


NOTES:
Congratuations! You've successfully installed Ambassador.

For help, visit our Slack at https://d6e.co/slack or view the documentation online at https://www.getambassador.io.

To get the IP address of Ambassador, run the following commands:
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
     You can watch the status of by running 'kubectl get svc -w  --namespace seldon ambassador'

  On GKE/Azure:
  export SERVICE_IP=$(kubectl get svc --namespace seldon ambassador -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

  On AWS:
  export SERVICE_IP=$(kubectl get svc --namespace seldon ambassador -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

  echo http://$SERVICE_IP:

[6]:
!kubectl rollout status deployment.apps/ambassador
Waiting for deployment "ambassador" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "ambassador" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "ambassador" rollout to finish: 2 of 3 updated replicas are available...
deployment "ambassador" successfully rolled out

Port forward to Ambassador

kubectl port-forward $(kubectl get pods -n seldon -l app.kubernetes.io/name=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8003:8080

Illustration of both REST and gRPC requests.

[7]:
!pygmentize resources/model_short_timeouts.json
{
    "apiVersion": "machinelearning.seldon.io/v1alpha2",
    "kind": "SeldonDeployment",
    "metadata": {
        "labels": {
            "app": "seldon"
        },
        "name": "seldon-deployment-example"
    },
    "spec": {
        "annotations": {
            "project_name": "FX Market Prediction",
            "deployment_version": "v1",
            "seldon.io/rest-read-timeout":"1"
        },
        "name": "test-deployment",
        "oauth_key": "oauth-key",
        "oauth_secret": "oauth-secret",
        "predictors": [
            {
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/mock_classifier:1.0",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier",
                                "resources": {
                                    "requests": {
                                        "memory": "1Mi"
                                    }
                                }
                            }
                        ],
                        "terminationGracePeriodSeconds": 20
                    }
                }],
                "graph": {
                    "children": [],
                    "name": "classifier",
                    "endpoint": {
                        "type" : "REST"
                    },
                    "type": "MODEL"
                },
                "name": "fx-market-predictor",
                "replicas": 1,
                "annotations": {
                    "predictor_version" : "v1"
                }
            }
        ]
    }
}

Seldon Deployment with Short Timeout

Deploy the runtime graph to kubernetes.

[8]:
!kubectl apply -f resources/model_short_timeouts.json -n seldon
seldondeployment.machinelearning.seldon.io/seldon-deployment-example created
[9]:
!kubectl rollout status deploy/test-deployment-fx-market-predictor-7cd068f
Waiting for deployment "test-deployment-fx-market-predictor-7cd068f" rollout to finish: 0 of 1 updated replicas are available...
deployment "test-deployment-fx-market-predictor-7cd068f" successfully rolled out
[10]:
from seldon_core.seldon_client import SeldonClient
sc = SeldonClient(deployment_name="seldon-deployment-example",namespace="seldon")

This next request should fail as the timeout is too short

[11]:
r = sc.predict(gateway="ambassador",transport="rest")
print(r)
Success:False message:504:Gateway Timeout
Request:
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.30659804050371386
  }
}

Response:
None

Delete this graph and recreate one with a longer timeout

[12]:
!kubectl delete -f resources/model_short_timeouts.json
seldondeployment.machinelearning.seldon.io "seldon-deployment-example" deleted

Seldon Deployment with Long Timeout

[13]:
!pygmentize resources/model_long_timeouts.json
{
    "apiVersion": "machinelearning.seldon.io/v1alpha2",
    "kind": "SeldonDeployment",
    "metadata": {
        "labels": {
            "app": "seldon"
        },
        "name": "model-long-timeout"
    },
    "spec": {
        "annotations": {
            "deployment_version": "v1",
            "seldon.io/rest-read-timeout":"100000",
            "seldon.io/rest-connection-timeout":"100000",
            "seldon.io/grpc-read-timeout":"100000"
        },
        "name": "long-to",
        "oauth_key": "oauth-key",
        "oauth_secret": "oauth-secret",
        "predictors": [
            {
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/mock_classifier:1.0",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier",
                                "resources": {
                                    "requests": {
                                        "memory": "1Mi"
                                    }
                                }
                            }
                        ],
                        "terminationGracePeriodSeconds": 20
                    }
                }],
                "graph": {
                    "children": [],
                    "name": "classifier",
                    "endpoint": {
                        "type" : "REST"
                    },
                    "type": "MODEL"
                },
                "name": "test",
                "replicas": 1,
                "annotations": {
                    "predictor_version" : "v1"
                }
            }
        ]
    }
}
[14]:
!kubectl apply -f resources/model_long_timeouts.json -n seldon
seldondeployment.machinelearning.seldon.io/model-long-timeout created
[15]:
!kubectl rollout status deploy/long-to-test-7cd068f
Waiting for deployment "long-to-test-7cd068f" rollout to finish: 0 of 1 updated replicas are available...
deployment "long-to-test-7cd068f" successfully rolled out

This next request should work as the timeout is much longer

[16]:
r = sc.predict(gateway="ambassador",transport="rest",deployment_name="model-long-timeout")
print(r)
Success:True message:
Request:
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.09274938985066539
  }
}

Response:
meta {
  puid: "s7flusof0k8hmv04ub97dqt68o"
  requestPath {
    key: "classifier"
    value: "seldonio/mock_classifier:1.0"
  }
}
data {
  names: "proba"
  tensor {
    shape: 1
    shape: 1
    values: 0.05604533219697675
  }
}

[ ]: