This page was generated from notebooks/timeouts.ipynb.
Testing Custom Timeouts¶
Prerequisites¶
Kubernetes cluster with kubectl authorized
curl and grpcurl installed
Setup Seldon Core¶
Use the setup notebook to Setup Cluster to setup Seldon Core with an ingress - either Ambassador or Istio.
Then port-forward to that ingress on localhost:8003 in a separate terminal either with:
Ambassador:
kubectl port-forward $(kubectl get pods -n seldon -l app.kubernetes.io/name=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8003:8080
Istio:
kubectl port-forward $(kubectl get pods -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}') -n istio-system 8003:8080
[ ]:
!kubectl create namespace seldon
[ ]:
!kubectl config set-context $(kubectl config current-context) --namespace=seldon
[ ]:
from IPython.core.magic import register_line_cell_magic
@register_line_cell_magic
def writetemplate(line, cell):
print("writing template")
with open(line, "w") as f:
f.write(cell.format(**globals()))
[ ]:
VERSION = !cat ../version.txt
VERSION = VERSION[0]
VERSION
Configure Istio¶
For this example we will create the default istio gateway for seldon which needs to be called seldon-gateway
. You can supply your own gateway by adding to your SeldonDeployments resources the annotation seldon.io/istio-gateway
with values the name of your istio gateway.
[ ]:
%%writefile resources/seldon-gateway.yaml
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: seldon-gateway
namespace: istio-system
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
[ ]:
!kubectl create -f resources/seldon-gateway.yaml -n istio-system
Short timeouts deployment file¶
Below is the outputs of the Seldon config file required to set custom timeouts.
First we’ll show how we can set a short lived example by setting the "seldon.io/rest-timeout":"1000"
annotation with a fake model that delays for 30 secs.
[ ]:
%%writetemplate resources/model_short_timeouts.json
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
name: seldon-model
spec:
annotations:
seldon.io/rest-timeout: "1000"
seldon.io/grpc-timeout: "1000"
predictors:
- componentSpecs:
- spec:
containers:
- image: seldonio/mock_classifier:{VERSION}
name: classifier
graph:
children: []
name: classifier
type: MODEL
parameters:
- name: delaySecs
type: INT
value: "30"
name: example
replicas: 1
We can then apply this Seldon Deployment file we defined above in the seldon
namespace.
[ ]:
!kubectl apply -f resources/model_short_timeouts.json -n seldon
And wait until the resource runs correctly.
[ ]:
!curl -i -d '{"data": {"ndarray":[[1.0, 2.0, 5.0]]}}' -X POST http://localhost:8003/seldon/seldon/seldon-model/api/v1.0/predictions -H "Content-Type: application/json"
[ ]:
!cd ../executor/proto && grpcurl -d '{"data":{"ndarray":[[1.0,2.0,5.0]]}}' \
-rpc-header seldon:seldon-model -rpc-header namespace:seldon \
-plaintext \
-proto ./prediction.proto 0.0.0.0:8003 seldon.protos.Seldon/Predict
Delete this graph and recreate one with a longer timeout
[ ]:
!kubectl delete -f resources/model_short_timeouts.json
Seldon Deployment with Long Timeout¶
Now we can have a look at how we can set a deployment with a longer timeout.
[ ]:
%%writetemplate resources/model_long_timeouts.json
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
name: seldon-model
spec:
annotations:
seldon.io/rest-timeout: "40000"
seldon.io/grpc-timeout: "40000"
predictors:
- componentSpecs:
- spec:
containers:
- image: seldonio/mock_classifier:{VERSION}
name: classifier
graph:
children: []
name: classifier
type: MODEL
parameters:
- name: delaySecs
type: INT
value: "30"
name: example
replicas: 1
And we can apply it to deploy the model with long timeouts.
[ ]:
!kubectl apply -f resources/model_long_timeouts.json -n seldon
[ ]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=seldon-model -o jsonpath='{.items[0].metadata.name}')
This next request will work given that the timeout is much longer.
[ ]:
!curl -i -d '{"data": {"ndarray":[[1.0, 2.0, 5.0]]}}' -X POST http://localhost:8003/seldon/seldon/seldon-model/api/v1.0/predictions -H "Content-Type: application/json"
[ ]:
!cd ../executor/proto && grpcurl -d '{"data":{"ndarray":[[1.0,2.0,5.0]]}}' \
-rpc-header seldon:seldon-model -rpc-header namespace:seldon \
-plaintext \
-proto ./prediction.proto 0.0.0.0:8003 seldon.protos.Seldon/Predict
[ ]:
!kubectl delete -f resources/model_long_timeouts.json