OpenVINO example with Squeezenet Model

This notebook illustrates how you can serve OpenVINO optimized models for Imagenet with Seldon Core.

car

Prerequisites:

  • pip install seldon-core

To run all of the notebook successfully you will need to start it with

jupyter notebook --NotebookApp.iopub_data_rate_limit=100000000

Download Squeezenet Model

We will download a pre-trained and optimized model for OpenVINO CPU into a local folder.

[1]:
!mkdir -p models/squeezenet/1 && \
    wget -O models/squeezenet/1/squeezenet1.1.xml https://s3-eu-west-1.amazonaws.com/seldon-public/openvino-squeeznet-model/squeezenet1.1.xml && \
    wget -O models/squeezenet/1/squeezenet1.1.mapping https://s3-eu-west-1.amazonaws.com/seldon-public/openvino-squeeznet-model/squeezenet1.1.mapping && \
    wget -O models/squeezenet/1/squeezenet1.1.bin https://s3-eu-west-1.amazonaws.com/seldon-public/openvino-squeeznet-model/squeezenet1.1.bin
--2019-04-24 13:28:04--  https://s3-eu-west-1.amazonaws.com/seldon-public/openvino-squeeznet-model/squeezenet1.1.xml
Resolving s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)... 52.218.105.2
Connecting to s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)|52.218.105.2|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 37345 (36K) [text/xml]
Saving to: ‘models/squeezenet/1/squeezenet1.1.xml’

models/squeezenet/1 100%[===================>]  36.47K  --.-KB/s    in 0.01s

2019-04-24 13:28:04 (2.53 MB/s) - ‘models/squeezenet/1/squeezenet1.1.xml’ saved [37345/37345]

--2019-04-24 13:28:04--  https://s3-eu-west-1.amazonaws.com/seldon-public/openvino-squeeznet-model/squeezenet1.1.mapping
Resolving s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)... 52.218.105.2
Connecting to s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)|52.218.105.2|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9318 (9.1K) [binary/octet-stream]
Saving to: ‘models/squeezenet/1/squeezenet1.1.mapping’

models/squeezenet/1 100%[===================>]   9.10K  --.-KB/s    in 0.001s

2019-04-24 13:28:04 (12.8 MB/s) - ‘models/squeezenet/1/squeezenet1.1.mapping’ saved [9318/9318]

--2019-04-24 13:28:04--  https://s3-eu-west-1.amazonaws.com/seldon-public/openvino-squeeznet-model/squeezenet1.1.bin
Resolving s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)... 52.218.105.2
Connecting to s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)|52.218.105.2|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4941984 (4.7M) [application/octet-stream]
Saving to: ‘models/squeezenet/1/squeezenet1.1.bin’

models/squeezenet/1 100%[===================>]   4.71M  10.5MB/s    in 0.5s

2019-04-24 13:28:04 (10.5 MB/s) - ‘models/squeezenet/1/squeezenet1.1.bin’ saved [4941984/4941984]

Run Seldon Core on Minikube

The example below assumes Minikube 0.30.0 installed

[2]:
!minikube start --memory 4096 --disk-size 20g --extra-config=apiserver.authorization-mode=RBAC
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
Kubectl is now configured to use the cluster.
Loading cached images from config file.

Setup Seldon Core

Use the setup notebook to Setup Cluster with Ambassador Ingress and Install Seldon Core. Instructions also online.

Mount local folder onto minikube for HostPath

Run in the current folder:

minikube mount ./models:/opt/ml

This will allow the model folder containing the Squeezenet model to be accessed. For production deployments you would use a NFS volume.

Deploy Seldon Intel OpenVINO Graph

[12]:
!helm install openvino-squeezenet ../../../helm-charts/seldon-openvino \
    --set openvino.model.path=/opt/ml/squeezenet \
    --set openvino.model.name=squeezenet1.1 \
    --set openvino.model.input=data \
    --set openvino.model.output=prob
NAME:   openvino-squeezenet
LAST DEPLOYED: Wed Apr 24 14:34:01 2019
NAMESPACE: seldon
STATUS: DEPLOYED

RESOURCES:
==> v1/PersistentVolume
NAME          CAPACITY  ACCESS MODES  RECLAIM POLICY  STATUS  CLAIM                   STORAGECLASS  REASON  AGE
hostpath-pvc  1Gi       RWO           Retain          Bound   seldon/model-store-pvc  manual        1s

==> v1/PersistentVolumeClaim
NAME             STATUS  VOLUME        CAPACITY  ACCESS MODES  STORAGECLASS  AGE
model-store-pvc  Bound   hostpath-pvc  1Gi       RWO           manual        1s

==> v1alpha2/SeldonDeployment
NAME            AGE
openvino-model  1s


[13]:
!helm template openvino-squeezenet ../../../helm-charts/seldon-openvino \
    --set openvino.model.path=/opt/ml/squeezenet \
    --set openvino.model.name=squeezenet1.1 \
    --set openvino.model.input=data \
    --set openvino.model.output=prob | pygmentize -l json
---
# Source: seldon-openvino/templates/hostPath.json

{
    "kind": "PersistentVolume",
    "apiVersion": "v1",
    "metadata": {
        "name": "hostpath-pvc"
    },
    "spec": {
        "capacity": {
            "storage": "1Gi"
        },
        "hostPath": {
            "path": "/opt/ml",
            "type": ""
        },
        "accessModes": [
            "ReadWriteOnce"
        ],
        "persistentVolumeReclaimPolicy": "Retain",
        "storageClassName": "manual"
    }
}

---
# Source: seldon-openvino/templates/openvino_deployment.json
{
    "apiVersion": "machinelearning.seldon.io/v1alpha2",
    "kind": "SeldonDeployment",
    "metadata": {
        "labels": {
            "app": "seldon"
        },
        "name": "openvino-model",
        "namespace": "seldon"
    },
    "spec": {
        "name": "openvino",
        "predictors": [
            {
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/tfserving-proxy:0.2",
                                "name": "tfserving-proxy"
                            },
                            {
                        "name": "openvino-model-server",
                        "image": "intelaipg/openvino-model-server:0.2",
                        "command": [
                            "/ie-serving-py/start_server.sh"
                        ],
                        "args": [
                            "ie_serving",
                            "model",
                            "--model_path",
                            "/opt/ml/squeezenet",
                            "--model_name",
                            "squeezenet1.1",
                            "--port",
                            "8001"
                        ],
                        "ports": [
                            {
                                "name": "grpc",
                                "containerPort": 8001,
                                "protocol": "TCP"
                            }
                        ],
                        "env": [
                            {
                                "name": "LOG_LEVEL",
                                "value": "DEBUG"
                            }
                        ],
                        "resources": {},
                        "volumeMounts": [
                            {
                                "name": "modelstore",
                                "mountPath": "/opt/ml"
                            }
                        ]
                            }
                        ],
                        "terminationGracePeriodSeconds": 1,
                        "volumes": [
                            {
                                "name": "modelstore",
                                "persistentVolumeClaim": {
                                    "claimName": "model-store-pvc"
                                }
                            }
                        ]
                    }
                }],
                "graph": {
                    "name": "tfserving-proxy",
                    "endpoint": { "type" : "GRPC" },
                    "type": "MODEL",
                    "children": [],
                    "parameters":
                    [
                        {
                            "name":"grpc_endpoint",
                            "type":"STRING",
                            "value":"localhost:8001"
                        },
                        {
                            "name":"model_name",
                            "type":"STRING",
                            "value":"squeezenet1.1"
                        },
                        {
                            "name":"model_output",
                            "type":"STRING",
                            "value":"prob"
                        },
                        {
                            "name":"model_input",
                            "type":"STRING",
                            "value":"data"
                        }
                    ]
                },
                "name": "openvino",
                "replicas": 1
            }
        ]
    }
}

---
# Source: seldon-openvino/templates/pvc.json
{
    "kind": "PersistentVolumeClaim",
    "apiVersion": "v1",
    "metadata": {
        "name": "model-store-pvc"
    },
    "spec": {
        "accessModes": [
            "ReadWriteOnce"
        ],
        "resources": {
            "requests": {
                "storage": "1Gi"
            }
        },
        "storageClassName": "manual"
    }
}
[15]:
!kubectl rollout status deploy/openvino-openvino-9740a00
deployment "openvino-openvino-9740a00" successfully rolled out

Serve Requests

Ensure you port forward ambassador:

kubectl port-forward $(kubectl get pods -n seldon -l app.kubernetes.io/name=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8003:8080
[17]:
%matplotlib inline
import numpy as np
from keras.applications.imagenet_utils import preprocess_input, decode_predictions
from keras.preprocessing import image
import sys
import json
import matplotlib.pyplot as plt
from seldon_core.seldon_client import SeldonClient

def getImage(path):
    img = image.load_img(path, target_size=(227, 227))
    x = image.img_to_array(img)
    plt.imshow(x/255.)
    x = np.expand_dims(x, axis=0)
    x = preprocess_input(x)
    return x

X = getImage("car.png")
X = X.transpose((0,3,1,2))
print(X.shape)

sc = SeldonClient(deployment_name="openvino-model",namespace="seldon")

response = sc.predict(gateway="ambassador",transport="rest",data=X)

result = response.response.data.tensor.values

result = np.array(result)
result = result.reshape(1,1000)

with open('imagenet_classes.json') as f:
    cnames = eval(f.read())

    for i in range(result.shape[0]):
        single_result = result[[i],...]
        ma = np.argmax(single_result)
        print("\t",i, cnames[ma])

(1, 3, 227, 227)
         0 sports car, sport car
../_images/examples_openvino_12_1.png
[ ]: