This page was generated from examples/ambassador/circuit_breakers/ambassador_circuit_breakers.ipynb.

Circuit Breakers with Seldon and Ambassador

This notebook shows how you can deploy Seldon Deployments which can have circuit breakers via Ambassador’s circuit breakers configuration.

Setup Seldon Core

Use the setup notebook to Setup Cluster with Ambassador Ingress and Install Seldon Core. Instructions also online.

!kubectl create namespace seldon

Error from server (AlreadyExists): namespaces "seldon" already exists

!kubectl config set-context $(kubectl config current-context) --namespace=seldon

Context "kind-ansible" modified.

Launch main model

We will create a very simple Seldon Deployment with a dummy model image seldonio/mock_classifier:1.0. This deployment is named example. We will add following circuit breakers configurations.

"seldon.io/ambassador-circuit-breakers-max-connections":"200",
"seldon.io/ambassador-circuit-breakers-max-pending-requests":"100",
"seldon.io/ambassador-circuit-breakers-max-requests":"200",
"seldon.io/ambassador-circuit-breakers-max-retries":"3"

Where

• "seldon.io/ambassador-circuit-breakers-max-connections":"200" is the maximum number of connections will make to the Seldon Deployment

• "seldon.io/ambassador-circuit-breakers-max-pending-requests":"100" is the maximum number of requests that will be queued while waiting for a connection

• "seldon.io/ambassador-circuit-breakers-max-requests":"200" is the maximum number of parallel outstanding requests to the Seldon Deployment

• "seldon.io/ambassador-circuit-breakers-max-retries":"3" the maximum number of parallel retries allowed to the Seldon Deployment

!pygmentize model_circuit_breakers_ambassador.json

{
    "apiVersion": "machinelearning.seldon.io/v1alpha2",
    "kind": "SeldonDeployment",
    "metadata": {
        "labels": {
            "app": "seldon"
        },
        "name": "example"
    },
    "spec": {
        "name": "production-model",
    "annotations": {
        "seldon.io/ambassador-circuit-breakers-max-connections":"200",
        "seldon.io/ambassador-circuit-breakers-max-pending-requests":"100",
        "seldon.io/ambassador-circuit-breakers-max-requests":"200",
        "seldon.io/ambassador-circuit-breakers-max-retries":"3"
    },
        "predictors": [
            {
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/mock_classifier_rest:1.4",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier"
                            }
                        ],
                        "terminationGracePeriodSeconds": 1
                    }}
                                  ],
                "graph":
                {
                    "children": [],
                    "name": "classifier",
                    "type": "MODEL",
                    "endpoint": {
                        "type": "REST"
                    }},
                "name": "single",
                "replicas": 1
            }
        ]
    }
}

!kubectl create -f model_circuit_breakers_ambassador.json

seldondeployment.machinelearning.seldon.io/example created

!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=example -o jsonpath='{.items[0].metadata.name}')

deployment "example-single-0-classifier" successfully rolled out

Get predictions

from seldon_core.seldon_client import SeldonClient

sc = SeldonClient(deployment_name="example", namespace="seldon")

2022-08-23 09:48:53.841515: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-08-23 09:48:53.841537: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.

REST Request

r = sc.predict(gateway="ambassador", transport="rest")
assert r.success == True
print(r)

Success:True message:
Request:
meta {
}
data {
  tensor {
    shape: 1
    shape: 1
    values: 0.28811053211964466
  }
}

Response:
{'data': {'names': ['proba'], 'tensor': {'shape': [1, 1], 'values': [0.06732305523577842]}}, 'meta': {}}

!kubectl delete -f model_circuit_breakers_ambassador.json

seldondeployment.machinelearning.seldon.io "example" deleted