This page was generated from examples/mnist_triton_e2e/mnist-triton.ipynb.

Tensorflow MNIST model and Triton (e2e example)

Prerequisites

  • A kubernetes cluster with kubectl configured

  • rclone

  • curl

  • Poetry (optional)

Setup Seldon Core

Use the setup notebook to Setup Cluster with Ambassador Ingress and Install Seldon Core. Instructions also online.

We will assume that ambassador (or Istio) ingress is port-forwarded to localhost:8003

Setup MinIO

Use the provided notebook to install Minio in your cluster. Instructions also online.

We will assume that MinIO service is port-forwarded to localhost:8090

[1]:
%%writefile rclone.conf
[s3]
type = s3
provider = minio
env_auth = false
access_key_id = admin@seldon.io
secret_access_key = 12341234
endpoint = http://localhost:8090
Overwriting rclone.conf
[2]:
%%writefile secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: seldon-rclone-secret
type: Opaque
stringData:
  RCLONE_CONFIG_S3_TYPE: s3
  RCLONE_CONFIG_S3_PROVIDER: minio
  RCLONE_CONFIG_S3_ENV_AUTH: "false"
  RCLONE_CONFIG_S3_ACCESS_KEY_ID: "admin@seldon.io"
  RCLONE_CONFIG_S3_SECRET_ACCESS_KEY: "12341234"
  RCLONE_CONFIG_S3_ENDPOINT: http://minio.minio-system.svc.cluster.local:9000
Overwriting secret.yaml
[3]:
!kubectl apply -f secret.yaml
secret/seldon-rclone-secret configured

Poetry (optional)

We will use poetry.lock to fully define the training environment. Install poetry following official documentation. Usually this goes down to

curl -sSL https://install.python-poetry.org | python3 - --version 1.1.15

Note: as runtime environment will be defined by the Triton Server image this step is optional. However, keeping the training environment fully reproducible is usually a right thing to do.

Train MNIST Model

Prepare training environment

[4]:
!conda create --yes --prefix ./venv python=3.8
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/rskolasinski/work/seldon-core-mnist-triton/examples/mnist_triton_e2e/venv

  added / updated specs:
    - conda-ecosystem-user-package-isolation
    - python=3.8


The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-1_gnu
  ca-certificates    conda-forge/linux-64::ca-certificates-2021.10.8-ha878542_0
  conda-ecosystem-u~ conda-forge/linux-64::conda-ecosystem-user-package-isolation-1.0-0
  ld_impl_linux-64   conda-forge/linux-64::ld_impl_linux-64-2.36.1-hea4e1c9_2
  libffi             conda-forge/linux-64::libffi-3.4.2-h9c3ff4c_4
  libgcc-ng          conda-forge/linux-64::libgcc-ng-11.2.0-h1d223b6_11
  libgomp            conda-forge/linux-64::libgomp-11.2.0-h1d223b6_11
  libnsl             conda-forge/linux-64::libnsl-2.0.0-h7f98852_0
  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-11.2.0-he4da1e4_11
  libzlib            conda-forge/linux-64::libzlib-1.2.11-h36c2ea0_1013
  ncurses            conda-forge/linux-64::ncurses-6.2-h58526e2_4
  openssl            conda-forge/linux-64::openssl-3.0.0-h7f98852_2
  pip                conda-forge/noarch::pip-21.3.1-pyhd8ed1ab_0
  python             conda-forge/linux-64::python-3.8.12-hf930737_2_cpython
  python_abi         conda-forge/linux-64::python_abi-3.8-2_cp38
  readline           conda-forge/linux-64::readline-8.1-h46c0cb4_0
  setuptools         conda-forge/linux-64::setuptools-58.5.3-py38h578d9bd_0
  sqlite             conda-forge/linux-64::sqlite-3.36.0-h9cd32fc_2
  tk                 conda-forge/linux-64::tk-8.6.11-h27826a3_1
  wheel              conda-forge/noarch::wheel-0.37.0-pyhd8ed1ab_1
  xz                 conda-forge/linux-64::xz-5.2.5-h516909a_1
  zlib               conda-forge/linux-64::zlib-1.2.11-h36c2ea0_1013


Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate /home/rskolasinski/work/seldon-core-mnist-triton/examples/mnist_triton_e2e/venv
#
# To deactivate an active environment, use
#
#     $ conda deactivate

[5]:
%%writefile pyproject.toml
[tool.poetry]
name = "mnist_triton_e2e"
version = "0.1.0"
description = ""
authors = ["Seldon Technologies Ltd. <hello@seldon.io>"]
license = "Apache-2.0"

[tool.poetry.dependencies]
python = "~3.8"
tensorflow = "2.6.2"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
Overwriting pyproject.toml

On first run of poetry install command it will create a poetry.lock file. Every next run of the installation command will use the package version from the lock file.

[6]:
%%bash
source ~/miniconda3/etc/profile.d/conda.sh
conda activate ./venv
poetry install
Installing dependencies from lock file

Package operations: 37 installs, 0 updates, 0 removals

  • Installing certifi (2021.10.8)
  • Installing charset-normalizer (2.0.7)
  • Installing idna (3.3)
  • Installing pyasn1 (0.4.8)
  • Installing urllib3 (1.26.7)
  • Installing cachetools (4.2.4)
  • Installing oauthlib (3.1.1)
  • Installing pyasn1-modules (0.2.8)
  • Installing rsa (4.7.2)
  • Installing requests (2.26.0)
  • Installing six (1.15.0)
  • Installing google-auth (1.35.0)
  • Installing requests-oauthlib (1.3.0)
  • Installing absl-py (0.15.0)
  • Installing google-auth-oauthlib (0.4.6)
  • Installing grpcio (1.41.1)
  • Installing numpy (1.19.5)
  • Installing tensorboard-data-server (0.6.1)
  • Installing tensorboard-plugin-wit (1.8.0)
  • Installing protobuf (3.19.1)
  • Installing werkzeug (2.0.2)
  • Installing markdown (3.3.4)
  • Installing astunparse (1.6.3)
  • Installing clang (5.0)
  • Installing gast (0.4.0)
  • Installing google-pasta (0.2.0)
  • Installing keras (2.6.0)
  • Installing keras-preprocessing (1.1.2)
  • Installing opt-einsum (3.3.0)
  • Installing tensorboard (2.6.0)
  • Installing termcolor (1.1.0)
  • Installing h5py (3.1.0)
  • Installing flatbuffers (1.12)
  • Installing typing-extensions (3.7.4.3)
  • Installing tensorflow-estimator (2.6.0)
  • Installing wrapt (1.12.1)
  • Installing tensorflow (2.6.2)

Prepare Training Script

[7]:
%%writefile train.py

import numpy as np
np.random.seed(123)   # for reproducibility

from tensorflow.keras.layers import (Activation, Convolution2D, Dense, Dropout,
                                     Flatten, MaxPooling2D)
from tensorflow.keras.models import Sequential
from tensorflow.keras.utils import to_categorical

from tensorflow.keras.datasets import mnist


# Load pre-shuffled MNIST data into train and test sets
(X_train, y_train), (X_test, y_test) = mnist.load_data()

# Reshape data for Tensorflow
X_train = X_train.reshape(-1, 28, 28, 1)
X_test = X_test.reshape(-1, 28, 28, 1)

X_train = X_train.astype("float32")
X_test = X_test.astype("float32")
X_train /= 255
X_test /= 255

# Convert 1-dimensional class arrays to 10-dimensional class matrices
Y_train = to_categorical(y_train, 10)
Y_test = to_categorical(y_test, 10)

# define model
model = Sequential()

# declare input layer
model.add(Convolution2D(32, (3, 3), activation="relu", input_shape=(28, 28, 1)))

# add more layers
model.add(Convolution2D(32, (3, 3), activation="relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

# and even more layers
model.add(Flatten())
model.add(Dense(128, activation="relu"))
model.add(Dropout(0.5))
model.add(Dense(10, activation="softmax"))

model.summary()

# compile model and train
model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"])
model.fit(X_train, Y_train, epochs=4, batch_size=32, verbose=1)

score = model.evaluate(X_test, Y_test, verbose=0)
print("\n%s: %.2f%%" % (model.metrics_names[1], score[1] * 100))

# save model
model.save("models-repository/mnist/1/model.savedmodel")
Overwriting train.py

Train Model

[8]:
!./venv/bin/python3 train.py
2021-11-09 00:16:11.767552: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-09 00:16:11.774042: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-09 00:16:11.774341: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-09 00:16:11.774801: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-11-09 00:16:11.775503: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-09 00:16:11.775756: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-09 00:16:11.775996: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-09 00:16:12.252515: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-09 00:16:12.252816: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-09 00:16:12.253061: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-09 00:16:12.253297: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1356 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1650, pci bus id: 0000:01:00.0, compute capability: 7.5
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d (Conv2D)              (None, 26, 26, 32)        320
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 24, 24, 32)        9248
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 12, 12, 32)        0
_________________________________________________________________
dropout (Dropout)            (None, 12, 12, 32)        0
_________________________________________________________________
flatten (Flatten)            (None, 4608)              0
_________________________________________________________________
dense (Dense)                (None, 128)               589952
_________________________________________________________________
dropout_1 (Dropout)          (None, 128)               0
_________________________________________________________________
dense_1 (Dense)              (None, 10)                1290
=================================================================
Total params: 600,810
Trainable params: 600,810
Non-trainable params: 0
_________________________________________________________________
2021-11-09 00:16:12.553857: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
Epoch 1/4
2021-11-09 00:16:13.375537: I tensorflow/stream_executor/cuda/cuda_dnn.cc:369] Loaded cuDNN version 8101
1875/1875 [==============================] - 8s 3ms/step - loss: 0.2046 - accuracy: 0.9378
Epoch 2/4
1875/1875 [==============================] - 5s 3ms/step - loss: 0.0872 - accuracy: 0.9742
Epoch 3/4
1875/1875 [==============================] - 6s 3ms/step - loss: 0.0676 - accuracy: 0.9791
Epoch 4/4
1875/1875 [==============================] - 6s 3ms/step - loss: 0.0546 - accuracy: 0.9833

accuracy: 98.98%
2021-11-09 00:16:38.051127: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.

Deploy and Test

Configure Triton Server

[9]:
%%writefile models-repository/mnist/config.pbtxt
name: "mnist"
platform: "tensorflow_savedmodel"
max_batch_size: 100
dynamic_batching { preferred_batch_size: [ 50 ]}
instance_group [ { count: 2 }]
input [
  {
    name: "conv2d_input"
    data_type: TYPE_FP32
    dims: [ 28, 28, 1 ]
  }
]
output [
  {
    name: "dense_1"
    data_type: TYPE_FP32
    dims: [ 10 ]
  }
]
Overwriting models-repository/mnist/config.pbtxt

Copy model into MinIO instance

[10]:
!tree models-repository/
models-repository/
└── mnist
    ├── 1
    │   └── model.savedmodel
    │       ├── assets
    │       ├── keras_metadata.pb
    │       ├── saved_model.pb
    │       └── variables
    │           ├── variables.data-00000-of-00001
    │           └── variables.index
    └── config.pbtxt

5 directories, 5 files
[11]:
!rclone -vvv --config="rclone.conf" copy models-repository s3://triton-models/mnist-model
2021/11/09 00:16:39 DEBUG : rclone: Version "v1.56.2" starting with parameters ["/home/rskolasinski/.asdf/installs/rclone/1.56.2/bin/rclone" "-vvv" "--config=rclone.conf" "copy" "models-repository" "s3://triton-models/mnist-model"]
2021/11/09 00:16:39 DEBUG : Creating backend with remote "models-repository"
2021/11/09 00:16:39 DEBUG : Using config file from "/home/rskolasinski/work/seldon-core-mnist-triton/examples/mnist_triton_e2e/rclone.conf"
2021/11/09 00:16:39 DEBUG : fs cache: renaming cache item "models-repository" to be canonical "/home/rskolasinski/work/seldon-core-mnist-triton/examples/mnist_triton_e2e/models-repository"
2021/11/09 00:16:39 DEBUG : Creating backend with remote "s3://triton-models/mnist-model"
2021/11/09 00:16:39 DEBUG : fs cache: renaming cache item "s3://triton-models/mnist-model" to be canonical "s3:triton-models/mnist-model"
2021/11/09 00:16:39 DEBUG : mnist/config.pbtxt: Modification times differ by -8m54.901977129s: 2021-11-09 00:16:39.278686721 +0000 GMT, 2021-11-09 00:07:44.376709592 +0000 GMT
2021/11/09 00:16:39 DEBUG : mnist/config.pbtxt: md5 = 8bf5c24a18dff3603cf2cdcbd5d5e929 OK
2021/11/09 00:16:39 DEBUG : S3 bucket triton-models path mnist-model: Waiting for checks to finish
2021/11/09 00:16:39 INFO  : mnist/config.pbtxt: Updated modification time in destination
2021/11/09 00:16:39 DEBUG : mnist/config.pbtxt: Unchanged skipping
2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/keras_metadata.pb: Modification times differ by -54m41.948759938s: 2021-11-09 00:16:38.51069757 +0000 GMT, 2021-11-08 23:21:56.561937632 +0000 GMT
2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/saved_model.pb: Modification times differ by -54m41.948760136s: 2021-11-09 00:16:38.49869774 +0000 GMT, 2021-11-08 23:21:56.549937604 +0000 GMT
2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/keras_metadata.pb: md5 = a4c5251702f32808e94e0d2b89a85aec OK
2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/variables/variables.index: Modification times differ by -54m41.956760288s: 2021-11-09 00:16:38.490697853 +0000 GMT, 2021-11-08 23:21:56.533937565 +0000 GMT
2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/variables/variables.index: md5 = 6e5f3d005dbe11b90507f2865a1a0713 (Local file system at /home/rskolasinski/work/seldon-core-mnist-triton/examples/mnist_triton_e2e/models-repository)
2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/variables/variables.index: md5 = be8df29950bcde41116269ed7b7ac6a2 (S3 bucket triton-models path mnist-model)
2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/variables/variables.index: md5 differ
2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/saved_model.pb: md5 = 936afcc716e6828695b1a11ad5a0d691 (Local file system at /home/rskolasinski/work/seldon-core-mnist-triton/examples/mnist_triton_e2e/models-repository)
2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/saved_model.pb: md5 = dcc8acd1b44e285751f93d4f3499decd (S3 bucket triton-models path mnist-model)
2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/saved_model.pb: md5 differ
2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/variables/variables.data-00000-of-00001: Modification times differ by -54m41.960760297s: 2021-11-09 00:16:38.490697853 +0000 GMT, 2021-11-08 23:21:56.529937556 +0000 GMT
2021/11/09 00:16:39 INFO  : mnist/1/model.savedmodel/keras_metadata.pb: Updated modification time in destination
2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/keras_metadata.pb: Unchanged skipping
2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/saved_model.pb: md5 = 936afcc716e6828695b1a11ad5a0d691 OK
2021/11/09 00:16:39 INFO  : mnist/1/model.savedmodel/saved_model.pb: Copied (replaced existing)
2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/variables/variables.index: md5 = 6e5f3d005dbe11b90507f2865a1a0713 OK
2021/11/09 00:16:39 INFO  : mnist/1/model.savedmodel/variables/variables.index: Copied (replaced existing)
2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/variables/variables.data-00000-of-00001: md5 = 254dd0a480879c42a9ec648b6f1fabf4 (Local file system at /home/rskolasinski/work/seldon-core-mnist-triton/examples/mnist_triton_e2e/models-repository)
2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/variables/variables.data-00000-of-00001: md5 = 176268afe35d0445761ce3c0698f4d16 (S3 bucket triton-models path mnist-model)
2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/variables/variables.data-00000-of-00001: md5 differ
2021/11/09 00:16:39 DEBUG : S3 bucket triton-models path mnist-model: Waiting for transfers to finish
2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/variables/variables.data-00000-of-00001: md5 = 254dd0a480879c42a9ec648b6f1fabf4 OK
2021/11/09 00:16:39 INFO  : mnist/1/model.savedmodel/variables/variables.data-00000-of-00001: Copied (replaced existing)
2021/11/09 00:16:39 INFO  :
Transferred:        7.026Mi / 7.026 MiByte, 100%, 0 Byte/s, ETA -
Checks:                 5 / 5, 100%
Transferred:            3 / 3, 100%
Elapsed time:         0.0s

2021/11/09 00:16:39 DEBUG : 15 go routines active

Deploy

[12]:
%%writefile deployment.yaml
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: mnist
  namespace: seldon
spec:
  name: default
  predictors:
  - graph:
      implementation: TRITON_SERVER
      logger:
        mode: all
      modelUri: s3:triton-models/mnist-model
      envSecretRefName: seldon-rclone-secret
      name: mnist
      type: MODEL
    name: default
    replicas: 1
  protocol: kfserving
Overwriting deployment.yaml
[13]:
!kubectl apply -f deployment.yaml
seldondeployment.machinelearning.seldon.io/mnist unchanged
[14]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=mnist -o jsonpath='{.items[0].metadata.name}')
deployment "mnist-default-0-mnist" successfully rolled out

Test (dummy data)

[15]:
import numpy as np
import requests
[16]:
!curl -s http://localhost:8003/seldon/seldon/mnist/v2/models/mnist | jq .
{
  "name": "mnist",
  "versions": [
    "1"
  ],
  "platform": "tensorflow_savedmodel",
  "inputs": [
    {
      "name": "conv2d_input",
      "datatype": "FP32",
      "shape": [
        -1,
        28,
        28,
        1
      ]
    }
  ],
  "outputs": [
    {
      "name": "dense_1",
      "datatype": "FP32",
      "shape": [
        -1,
        10
      ]
    }
  ]
}
[17]:
URL = "http://localhost:8003/seldon/seldon/mnist"


def predict(data):
    data = {
        "inputs": [
            {
                "name": "conv2d_input",
                "data": data.tolist(),
                "datatype": "FP32",
                "shape": data.shape,
            }
        ]
    }

    r = requests.post(f"{URL}/v2/models/mnist/infer", json=data)
    predictions = np.array(r.json()["outputs"][0]["data"]).reshape(
        r.json()["outputs"][0]["shape"]
    )
    output = [np.argmax(x) for x in predictions]
    return output
[18]:
output = predict(np.random.rand(10, 28, 28, 1))
output
[18]:
[8, 8, 8, 8, 2, 8, 8, 8, 8, 3]

Test (real data - optional)

[19]:
import matplotlib.pyplot as plt
from tensorflow.keras.datasets import mnist
from tensorflow.keras.utils import to_categorical
2021-11-09 00:16:41.193198: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
[20]:
(X_train, y_train), (X_test, y_test) = mnist.load_data()

X_test = X_test.reshape(-1, 28, 28, 1)
X_test = X_test.astype("float32")
X_test /= 255

Y_test = to_categorical(y_test, 10)
[21]:
predictions = predict(X_test[:5])
fig, axs = plt.subplots(1, 5, figsize=(15, 3))

for i in range(5):
    axs[i].imshow(X_test[i, :, :, 0], cmap="binary")
    axs[i].set_title("{} - should be {}".format(predictions[i], np.argmax(Y_test[i])))
../_images/examples_triton_mnist_e2e_35_0.png