OpenVINO example with Squeezenet Model

This notebook illustrates how you can serve OpenVINO optimized models for Imagenet with Seldon Core.

car

Prerequisites:

  • pip install seldon-core

To run all of the notebook successfully you will need to start it with

jupyter notebook --NotebookApp.iopub_data_rate_limit=100000000

Download Squeezenet Model

We will download a pre-trained and optimized model for OpenVINO CPU into a local folder.

[1]:
!mkdir -p models/squeezenet/1 && \
    wget -O models/squeezenet/1/squeezenet1.1.xml https://s3-eu-west-1.amazonaws.com/seldon-public/openvino-squeeznet-model/squeezenet1.1.xml && \
    wget -O models/squeezenet/1/squeezenet1.1.mapping https://s3-eu-west-1.amazonaws.com/seldon-public/openvino-squeeznet-model/squeezenet1.1.mapping && \
    wget -O models/squeezenet/1/squeezenet1.1.bin https://s3-eu-west-1.amazonaws.com/seldon-public/openvino-squeeznet-model/squeezenet1.1.bin
--2019-04-24 13:28:04--  https://s3-eu-west-1.amazonaws.com/seldon-public/openvino-squeeznet-model/squeezenet1.1.xml
Resolving s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)... 52.218.105.2
Connecting to s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)|52.218.105.2|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 37345 (36K) [text/xml]
Saving to: ‘models/squeezenet/1/squeezenet1.1.xml’

models/squeezenet/1 100%[===================>]  36.47K  --.-KB/s    in 0.01s

2019-04-24 13:28:04 (2.53 MB/s) - ‘models/squeezenet/1/squeezenet1.1.xml’ saved [37345/37345]

--2019-04-24 13:28:04--  https://s3-eu-west-1.amazonaws.com/seldon-public/openvino-squeeznet-model/squeezenet1.1.mapping
Resolving s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)... 52.218.105.2
Connecting to s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)|52.218.105.2|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9318 (9.1K) [binary/octet-stream]
Saving to: ‘models/squeezenet/1/squeezenet1.1.mapping’

models/squeezenet/1 100%[===================>]   9.10K  --.-KB/s    in 0.001s

2019-04-24 13:28:04 (12.8 MB/s) - ‘models/squeezenet/1/squeezenet1.1.mapping’ saved [9318/9318]

--2019-04-24 13:28:04--  https://s3-eu-west-1.amazonaws.com/seldon-public/openvino-squeeznet-model/squeezenet1.1.bin
Resolving s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)... 52.218.105.2
Connecting to s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)|52.218.105.2|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4941984 (4.7M) [application/octet-stream]
Saving to: ‘models/squeezenet/1/squeezenet1.1.bin’

models/squeezenet/1 100%[===================>]   4.71M  10.5MB/s    in 0.5s

2019-04-24 13:28:04 (10.5 MB/s) - ‘models/squeezenet/1/squeezenet1.1.bin’ saved [4941984/4941984]

Run Seldon Core on Minikube

The example below assumes Minikube 0.30.0 installed

[2]:
!minikube start --memory 4096 --disk-size 20g --extra-config=apiserver.authorization-mode=RBAC
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
Kubectl is now configured to use the cluster.
Loading cached images from config file.
[27]:
!kubectl create namespace seldon
namespace/seldon created
[28]:
!kubectl config set-context $(kubectl config current-context) --namespace=seldon
Context "minikube" modified.
[29]:
!kubectl create clusterrolebinding kube-system-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
clusterrolebinding.rbac.authorization.k8s.io/kube-system-cluster-admin created
[30]:
!helm init
$HELM_HOME has been configured at /home/clive/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!
[31]:
!kubectl rollout status deploy/tiller-deploy -n kube-system
Waiting for deployment "tiller-deploy" rollout to finish: 0 of 1 updated replicas are available...
deployment "tiller-deploy" successfully rolled out
[4]:
!helm install ../../../helm-charts/seldon-core-operator --name seldon-core --set usageMetrics.enabled=true --namespace seldon-system
NAME:   seldon-core
LAST DEPLOYED: Wed Apr 24 14:30:26 2019
NAMESPACE: seldon-system
STATUS: DEPLOYED

RESOURCES:
==> v1/StatefulSet
NAME                                DESIRED  CURRENT  AGE
seldon-operator-controller-manager  1        1        0s

==> v1/Pod(related)
NAME                                  READY  STATUS             RESTARTS  AGE
seldon-operator-controller-manager-0  0/1    ContainerCreating  0         0s

==> v1/Secret
NAME                                   TYPE    DATA  AGE
seldon-operator-webhook-server-secret  Opaque  0     0s

==> v1beta1/CustomResourceDefinition
NAME                                         AGE
seldondeployments.machinelearning.seldon.io  0s

==> v1/ClusterRole
seldon-operator-manager-role  0s

==> v1/ClusterRoleBinding
NAME                                 AGE
seldon-operator-manager-rolebinding  0s

==> v1/Service
NAME                                        TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)  AGE
seldon-operator-controller-manager-service  ClusterIP  10.106.190.234  <none>       443/TCP  0s


NOTES:
NOTES: TODO


[11]:
!kubectl rollout status statefulset.apps/seldon-operator-controller-manager -n seldon-system
partitioned roll out complete: 1 new pods have been updated...

Setup Ingress

Please note: There are reported gRPC issues with ambassador (see https://github.com/SeldonIO/seldon-core/issues/473).

[7]:
!helm install stable/ambassador --name ambassador --set crds.keep=false
NAME:   ambassador
LAST DEPLOYED: Wed Apr 24 14:30:56 2019
NAMESPACE: seldon
STATUS: DEPLOYED

RESOURCES:
==> v1/Service
NAME               TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)                     AGE
ambassador-admins  ClusterIP     10.96.174.103  <none>       8877/TCP                    0s
ambassador         LoadBalancer  10.103.89.121  <pending>    80:30996/TCP,443:31363/TCP  0s

==> v1/Deployment
NAME        DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
ambassador  3        3        3           0          0s

==> v1/Pod(related)
NAME                         READY  STATUS             RESTARTS  AGE
ambassador-5b89d44544-45vms  0/1    ContainerCreating  0         0s
ambassador-5b89d44544-c66rn  0/1    ContainerCreating  0         0s
ambassador-5b89d44544-c8p8k  0/1    Pending            0         0s

==> v1/ServiceAccount
NAME        SECRETS  AGE
ambassador  1        0s

==> v1beta1/ClusterRole
NAME        AGE
ambassador  0s

==> v1beta1/ClusterRoleBinding
NAME        AGE
ambassador  0s


NOTES:
Congratuations! You've successfully installed Ambassador.

For help, visit our Slack at https://d6e.co/slack or view the documentation online at https://www.getambassador.io.

To get the IP address of Ambassador, run the following commands:
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
     You can watch the status of by running 'kubectl get svc -w  --namespace seldon ambassador'

  On GKE/Azure:
  export SERVICE_IP=$(kubectl get svc --namespace seldon ambassador -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

  On AWS:
  export SERVICE_IP=$(kubectl get svc --namespace seldon ambassador -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

  echo http://$SERVICE_IP:

[10]:
!kubectl rollout status deployment.apps/ambassador
deployment "ambassador" successfully rolled out

Mount local folder onto minikube for HostPath

Run in the current folder:

minikube mount ./models:/opt/ml

This will allow the model folder containing the Squeezenet model to be accessed. For production deployments you would use a NFS volume.

Deploy Seldon Intel OpenVINO Graph

[12]:
!helm install ../../../helm-charts/seldon-openvino --name=openvino-squeezenet \
    --set openvino.model.path=/opt/ml/squeezenet \
    --set openvino.model.name=squeezenet1.1 \
    --set openvino.model.input=data \
    --set openvino.model.output=prob
NAME:   openvino-squeezenet
LAST DEPLOYED: Wed Apr 24 14:34:01 2019
NAMESPACE: seldon
STATUS: DEPLOYED

RESOURCES:
==> v1/PersistentVolume
NAME          CAPACITY  ACCESS MODES  RECLAIM POLICY  STATUS  CLAIM                   STORAGECLASS  REASON  AGE
hostpath-pvc  1Gi       RWO           Retain          Bound   seldon/model-store-pvc  manual        1s

==> v1/PersistentVolumeClaim
NAME             STATUS  VOLUME        CAPACITY  ACCESS MODES  STORAGECLASS  AGE
model-store-pvc  Bound   hostpath-pvc  1Gi       RWO           manual        1s

==> v1alpha2/SeldonDeployment
NAME            AGE
openvino-model  1s


[13]:
!helm template ../../../helm-charts/seldon-openvino --name=openvino-squeezenet \
    --set openvino.model.path=/opt/ml/squeezenet \
    --set openvino.model.name=squeezenet1.1 \
    --set openvino.model.input=data \
    --set openvino.model.output=prob | pygmentize -l json
---
# Source: seldon-openvino/templates/hostPath.json

{
    "kind": "PersistentVolume",
    "apiVersion": "v1",
    "metadata": {
        "name": "hostpath-pvc"
    },
    "spec": {
        "capacity": {
            "storage": "1Gi"
        },
        "hostPath": {
            "path": "/opt/ml",
            "type": ""
        },
        "accessModes": [
            "ReadWriteOnce"
        ],
        "persistentVolumeReclaimPolicy": "Retain",
        "storageClassName": "manual"
    }
}

---
# Source: seldon-openvino/templates/openvino_deployment.json
{
    "apiVersion": "machinelearning.seldon.io/v1alpha2",
    "kind": "SeldonDeployment",
    "metadata": {
        "labels": {
            "app": "seldon"
        },
        "name": "openvino-model",
        "namespace": "seldon"
    },
    "spec": {
        "name": "openvino",
        "predictors": [
            {
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/tfserving-proxy:0.2",
                                "name": "tfserving-proxy"
                            },
                            {
                        "name": "openvino-model-server",
                        "image": "intelaipg/openvino-model-server:0.2",
                        "command": [
                            "/ie-serving-py/start_server.sh"
                        ],
                        "args": [
                            "ie_serving",
                            "model",
                            "--model_path",
                            "/opt/ml/squeezenet",
                            "--model_name",
                            "squeezenet1.1",
                            "--port",
                            "8001"
                        ],
                        "ports": [
                            {
                                "name": "grpc",
                                "containerPort": 8001,
                                "protocol": "TCP"
                            }
                        ],
                        "env": [
                            {
                                "name": "LOG_LEVEL",
                                "value": "DEBUG"
                            }
                        ],
                        "resources": {},
                        "volumeMounts": [
                            {
                                "name": "modelstore",
                                "mountPath": "/opt/ml"
                            }
                        ]
                            }
                        ],
                        "terminationGracePeriodSeconds": 1,
                        "volumes": [
                            {
                                "name": "modelstore",
                                "persistentVolumeClaim": {
                                    "claimName": "model-store-pvc"
                                }
                            }
                        ]
                    }
                }],
                "graph": {
                    "name": "tfserving-proxy",
                    "endpoint": { "type" : "GRPC" },
                    "type": "MODEL",
                    "children": [],
                    "parameters":
                    [
                        {
                            "name":"grpc_endpoint",
                            "type":"STRING",
                            "value":"localhost:8001"
                        },
                        {
                            "name":"model_name",
                            "type":"STRING",
                            "value":"squeezenet1.1"
                        },
                        {
                            "name":"model_output",
                            "type":"STRING",
                            "value":"prob"
                        },
                        {
                            "name":"model_input",
                            "type":"STRING",
                            "value":"data"
                        }
                    ]
                },
                "name": "openvino",
                "replicas": 1
            }
        ]
    }
}

---
# Source: seldon-openvino/templates/pvc.json
{
    "kind": "PersistentVolumeClaim",
    "apiVersion": "v1",
    "metadata": {
        "name": "model-store-pvc"
    },
    "spec": {
        "accessModes": [
            "ReadWriteOnce"
        ],
        "resources": {
            "requests": {
                "storage": "1Gi"
            }
        },
        "storageClassName": "manual"
    }
}
[15]:
!kubectl rollout status deploy/openvino-openvino-9740a00
deployment "openvino-openvino-9740a00" successfully rolled out

Serve Requests

Ensure you port forward ambassador:

kubectl port-forward $(kubectl get pods -n seldon -l app.kubernetes.io/name=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8003:8080
[17]:
%matplotlib inline
import numpy as np
from keras.applications.imagenet_utils import preprocess_input, decode_predictions
from keras.preprocessing import image
import sys
import json
import matplotlib.pyplot as plt
from seldon_core.seldon_client import SeldonClient

def getImage(path):
    img = image.load_img(path, target_size=(227, 227))
    x = image.img_to_array(img)
    plt.imshow(x/255.)
    x = np.expand_dims(x, axis=0)
    x = preprocess_input(x)
    return x

X = getImage("car.png")
X = X.transpose((0,3,1,2))
print(X.shape)

sc = SeldonClient(deployment_name="openvino-model",namespace="seldon")

response = sc.predict(gateway="ambassador",transport="rest",data=X)

result = response.response.data.tensor.values

result = np.array(result)
result = result.reshape(1,1000)

with open('imagenet_classes.json') as f:
    cnames = eval(f.read())

    for i in range(result.shape[0]):
        single_result = result[[i],...]
        ma = np.argmax(single_result)
        print("\t",i, cnames[ma])

(1, 3, 227, 227)
         0 sports car, sport car
../_images/examples_openvino_21_1.png
[ ]: