This page was generated from examples/models/gpu_tensorflow_deep_mnist/gpu_tensorflow_deep_mnist.ipynb.
Tensorflow GPU MNIST Model with GKE¶
Please note: This tutorial uses Tensorflow-gpu=1.13.1, CUDA 10.0 and cuDNN 7.6
Requirements: Ubuntu 18.+ and Python 3.6
In this tutorial we will run a deep MNIST Tensorflow example with GPU.
The tutorial will be broken down into the following sections:
Install all dependencies to run Tensorflow-GPU
1.1 Installing CUDA 10.0
1.2 Installing cuDNN 7.6
1.3 Configure CUDA and cuDNN
1.4 Install Tensorflow GPU
Train the MNIST model locally
Push the Image to your proejcts Container Registry
Deploy the model on GKE using Seldon Core
Local Testing Environment¶
For the development of this example a GCE Virtual Machine was used to allow access to a GPU. The configuration for this VM is as follows:
VM Image: TensorFlow from NVIDIA
8 vCPUs
32 GB memory
1x NVIDIA Tesla V100 GPU
1) Installing all dependencies to run Tensorflow-GPU¶
Dependencies installed in this section:
Nvidia compute 3.0 onwards
CUDA 10.0
cuDNN 7.6
tensorflow-gpu 1.13.1
Check Nvidia drivers >= 3.0
[ ]:
!nvidia-smi
1.1) Install CUDA 10.0¶
Download the CUDA 10.0 runfile
[ ]:
!wget https://developer.nvidia.com/compute/cuda/10.0/Prod/local_installers/cuda_10.0.130_410.48_linux
Unpack the separate files:
[ ]:
! chmod +x cuda_10.0.130_410.48_linux
! ./cuda_10.0.130_410.48_linux --extract=$HOME
Install the Cuda 10.0 Toolkit file:
From the terminal, run the following command
$ sudo ./cuda-linux.10.0.130-24817639.run
Hold ‘d’ to scroll to the bottom of the license agreement.
Accept the licencing agreement and all of the default settings.
Verify the install, by installing the sample test:
$ sudo ./cuda-samples.10.0.130-24817639-linux.run
Again, accept the agreement and all default settings
Configure the runtime library:
$ sudo bash -c "echo /usr/local/cuda/lib64/ > /etc/ld.so.conf.d/cuda.conf"
$ sudo ldconfig
Add the cuda bin to the file system:
$ sudo vim /etc/environment
Add ‘:/usr/local/cuda/bin’ to the end of the PATH (inside quotes)
Reboot the system
[ ]:
!sudo shutdown -r now
Run the tests that we set up - this takes some time to complete, so let it run for a little while…
$ cd /usr/local/cuda-10.0/samples
$ sudo make
If run into an error involving the GCC version:
$ sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-6 10
$ sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-6 10
And run again, otherwise, skip this step.
After complete, run a devicequery and bandwidth test:
[ ]:
%%bash
cd /usr/local/cuda/samples/bin/x86_64/linux/release
./deviceQuery
Remember to clean up by removing all of the downloaded runtime packages
1.2) Install cuDNN 7.6¶
Download all 3 .deb files for CUDA10.0 and Ubuntu 18.04
You will have to create a Nvidia account for this and go to the archive section of the cuDNN downloads
Ensure you download all 3 files: - Runtime - Developer - Code Samples
Unpackage the three files in this order
[ ]:
%%bash
sudo dpkg -i ~/libcudnn7_7.6.0.64-1+cuda10.0_amd64.deb
sudo dpkg -i ~/libcudnn7-dev_7.6.0.64-1+cuda10.0_amd64.deb
sudo dpkg -i ~/libcudnn7-doc_7.6.0.64-1+cuda10.0_amd64.deb
Verify the install is successful with the MNIST example
From the download folder. Copy the files to somewhere with write access:
[ ]:
! cp -r /usr/src/cudnn_samples_v7/ ~
Go to the MNIST example code, compile and run it
[ ]:
%%bash
cd ~/cudnn_samples_v7/mnistCUDNN
sudo make
sudo ./mnistCUDNN
Remember to clean up by removing all of the downloaded runtime packages
1.3) Configure CUDA and cuDNN¶
Add LD_LIBRARY_PATH in your .bashrc file:
Add the following line in the end or your .bashrc file export export:
LD_LIBRARY_PATH=/usr/local/cuda/lib64:${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
And source it with:
$ source ~/.bashrc
1.4) Install tensorflow with GPU¶
Require v=1.13.1 as with CUDA 10.0
[ ]:
! pip3 install --upgrade tensorflow-gpu==1.13.1
[ ]:
import tensorflow as tf
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
2) Train the MNIST model locally¶
Wrap a Tensorflow MNIST python model for use as a prediction microservice in seldon-core
Run locally on Docker to test
Deploy on seldon-core running on minikube
Dependencies¶
pip3 install seldon-core
Train locally¶
[ ]:
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
import tensorflow as tf
if __name__ == "__main__":
x = tf.placeholder(tf.float32, [None, 784], name="x")
W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))
y = tf.nn.softmax(tf.matmul(x, W) + b, name="y")
y_ = tf.placeholder(tf.float32, [None, 10])
cross_entropy = tf.reduce_mean(
-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1])
)
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)
init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)
for i in range(1000):
batch_xs, batch_ys = mnist.train.next_batch(100)
sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})
correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
print(sess.run(accuracy, feed_dict={x: mnist.test.images, y_: mnist.test.labels}))
saver = tf.train.Saver()
saver.save(sess, "model/deep_mnist_model")
Wrap model using s2i
[ ]:
!s2i build . seldonio/seldon-core-s2i-python3-tf-gpu:0.1 deep-mnist-gpu:0.1
[ ]:
!docker run --name "mnist_predictor" -d --rm -p 5000:5000 deep-mnist-gpu:0.1
Send some random features that conform to the contract
[ ]:
!seldon-core-tester contract.json 0.0.0.0 5000 -p
[ ]:
!docker rm mnist_predictor --force
3) Push the image to Google Container Registry¶
Configure access to container registry (follow the configuration to link to your own project).
$ gcloud auth configure-docker
Tag Image with your project’s registry path (Edit the command below)
[ ]:
!docker tag deep-mnist-gpu:0.1 gcr.io/<YOUR_PROJECT_ID>/deep-mnist-gpu:0.1
Push the Image to the Container Registry (Again edit command below)
[ ]:
!docker push gcr.io/<YOUR_PROJECT_ID>/deep-mnist-gpu:0.1
4) Deploy in GKE¶
Spin up a GKE Cluster¶
For this example only one node is needed within the cluster. The cluster should have the following config:
8 CPUs
30 GB Total Memory
1 Node with 1X NVIDIA Tesla V100 GPU
Ubuntu Node image
Leave the rest of the config as default.
Connect to your cluster and check the context.
[ ]:
!gcloud config set project <YOUR_PROJECT_ID>
!gcloud container clusters get-credentials <YOUR_CLUSTER_NAME>
!kubectl config current-context
Installing NVIDIA GPU device drivers
(The below command is for the Ubuntu Node Image - if using a COS image, please see the Google Cloud Documentation for the correct command).
[ ]:
!kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/ubuntu/daemonset-preloaded.yaml
Setup Seldon Core¶
Use the setup notebook to Setup Cluster with Ambassador Ingress and Install Seldon Core. Instructions also online.
Build the Seldon Graph¶
First lets look at the Seldon Graph Yaml file:
[74]:
!cat deep_mnist_gpu.json
{
"apiVersion": "machinelearning.seldon.io/v1alpha2",
"kind": "SeldonDeployment",
"metadata": {
"labels": {
"app": "seldon"
},
"name": "deep-mnist-gpu"
},
"spec": {
"annotations": {
"project_name": "Tensorflow MNIST",
"deployment_version": "v1"
},
"name": "deep-mnist-gpu",
"predictors": [
{
"componentSpecs": [{
"spec": {
"containers": [
{
"image": "gcr.io/<YOUR_PROJECT_ID>/deep-mnist-gpu:0.1",
"imagePullPolicy": "IfNotPresent",
"name": "classifier",
"resources": {
"limits": {
"nvidia.com/gpu": 1
}
}
}
],
"terminationGracePeriodSeconds": 20
}
}],
"graph": {
"children": [],
"name": "classifier",
"endpoint": {
"type" : "REST"
},
"type": "MODEL"
},
"name": "single-model",
"replicas": 1,
"annotations": {
"predictor_version" : "v1"
}
}
]
}
}
Change the image name in this file (line 24) to match the path to the image in your container registry.
$vim deep_mnist_gpu.json
Next, we are ready to build the seldon graph.
[65]:
!kubectl create -f deep_mnist_gpu.json
seldondeployment.machinelearning.seldon.io/deep-mnist-gpu created
[15]:
!kubectl rollout status deploy/deep-mnist-gpu-single-model-8969cc0
Error from server (NotFound): deployments.extensions "deep-mnist-gpu-single-model-8969cc0" not found
Check the deployment is running
[71]:
!kubectl get pods
NAME READY STATUS RESTARTS AGE
ambassador-865c877494-2td9s 1/1 Running 0 101m
ambassador-865c877494-2vsk2 1/1 Running 0 101m
ambassador-865c877494-qzh4c 1/1 Running 0 101m
deep-mnist-gpu-single-model-0588ac2-865d745b7d-kqcp9 2/2 Running 0 71m
seldon-operator-controller-manager-0 1/1 Running 1 101m
Test the deployment with test data¶
Change the IP address to the External IP of your Ambassador deployment.
[72]:
!kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ambassador LoadBalancer 10.76.8.138 104.197.71.69 80:30783/TCP,443:32277/TCP 101m
ambassador-admins ClusterIP 10.76.12.144 <none> 8877/TCP 101m
deep-mnist-gpu-deep-mnist-gpu ClusterIP 10.76.5.205 <none> 8000/TCP,5001/TCP 71m
kubernetes ClusterIP 10.76.0.1 <none> 443/TCP 107m
seldon-87fe3957f4554e9b5af993717a0b9327 ClusterIP 10.76.14.160 <none> 9000/TCP 71m
seldon-operator-controller-manager-service ClusterIP 10.76.8.100 <none> 443/TCP 101m
webhook-server-service ClusterIP 10.76.7.151 <none> 443/TCP 101m
[73]:
!seldon-core-api-tester contract.json <EXTERNAL_IP_ADDRESS> `kubectl get svc ambassador -o jsonpath='{.spec.ports[0].port}'` \
deep-mnist-gpu --namespace default -p
----------------------------------------
SENDING NEW REQUEST:
[[0.798 0.827 0.034 0.384 0.938 0.036 0.135 0.555 0.86 0.263 0.411 0.894
0.327 0.865 0.906 0.914 0.133 0.565 0.803 0.417 0.825 0.678 0.805 0.206
0.017 0.698 0.41 0.503 0.984 0.214 0.468 0.366 0.132 0.973 0.472 0.346
0.001 0.662 0.412 0.537 0.522 0.242 0.289 0.676 0.379 0.542 0.452 0.467
0.392 1. 0.771 0.442 0.352 0.505 0.259 0.505 0.664 0.942 0.457 0.417
0.895 0.42 0.322 0.885 0.578 0.528 0.222 0.283 0.137 0.605 0.915 0.182
0.42 0.94 0.262 0.599 0.552 0.437 0.179 0.928 0.831 0.193 0.391 0.416
0.315 0.012 0.815 0.925 0.52 0.773 0.93 0.673 0.757 0.979 0.151 0.459
0.621 0.553 0.605 0.176 0.702 0.814 0.784 0.952 0.513 0.125 0.68 0.043
0.377 0.67 0.466 0.824 0.245 0.221 0.324 0.749 0.182 0.992 0.243 0.855
0.477 0.176 0.262 0.537 0.69 0.717 0.059 0.711 0.26 0.149 0.34 0.71
0.041 0.623 0.447 0.319 0.089 0.954 0.435 0.267 0.416 0.275 0.923 0.254
0.542 0.995 0.782 0.337 0.991 0.187 0.183 0.479 0.73 0.288 0.6 0.583
0.392 0.389 0.572 0.281 0.016 0.097 0.745 0.161 0.053 0.994 0.998 0.21
0.348 0.531 0.423 0.894 0.153 0.759 0.277 0.002 0.113 0.236 0.171 0.979
0.315 0.171 0.217 0.328 0.995 0.231 0.134 0.69 0.468 0.437 0.536 0.198
0.412 0.15 0.465 0.402 0.975 0.698 0.057 0.885 0.433 0.463 0.73 0.285
0.429 0.068 0.942 0.367 0.96 0.042 0.383 0.498 0.563 0.606 0.139 0.148
0.151 0.4 0.946 0.805 0.954 0.739 0.925 0.305 0.909 0.222 0.475 0.729
0.679 0.43 0.7 0.085 0.103 0.3 0.073 0.263 0.472 0.998 0.615 0.218
0.677 0.555 0.155 0.093 0.36 0.149 0.343 0.801 0.896 0.106 0.253 0.875
0.245 0.853 0.909 0.958 0.362 0.663 0.674 0.298 0.139 0.118 0.242 0.282
0.095 0.755 0.635 0.168 0.259 0.515 0.77 0.196 0.185 0.659 0.379 0.64
0.351 0.184 0.723 0.639 0.893 0.132 0.833 0.377 0.486 0.262 0.091 0.694
0.043 0.957 0.927 0.469 0.47 0.407 0.166 0.673 0.065 0.582 0.403 0.795
0.39 0.991 0.723 0.863 0.347 0.612 0.63 0.628 0.298 0.398 0.788 0.491
0.497 0.669 0.016 0.609 0.778 0.379 0.454 0.113 0.4 0.649 0.155 0.687
0.317 0.248 0.044 0.933 0.615 0.335 0.022 0.661 0.582 0.418 0.053 0.924
0.69 0.723 0.007 0.149 0.703 0.1 0.799 0.991 0.877 0.626 0.191 0.829
0.07 0.814 0.989 0.664 0.192 0.849 0.611 0.78 0.397 0.281 0.688 0.876
0.423 0.185 0.036 0.476 0.417 0.804 0.336 0.498 0.653 0.585 0.339 0.155
0.438 0.781 0.321 0.462 0.595 0.324 0.463 0.065 0.655 0.534 0.01 0.906
0.836 0.389 0.457 0.629 0.831 0.145 0.082 0.889 0.231 0.075 0.404 0.408
0.035 0.226 0.371 0.961 0.907 0.366 0.937 0.818 0.373 0.813 0.645 0.009
0.16 0.797 0.81 0.48 0.76 0.464 0.127 0.842 0.531 0.362 0.546 0.95
0.788 0.069 0.276 0.79 0.287 0.64 0.797 0.262 0.132 0.317 0.766 0.759
0.714 0.642 0.601 0.482 0.529 0.43 0.934 0.07 0.137 0.794 0.5 0.065
0.157 0.672 0.858 0.336 0.991 0.054 0.352 0.163 0.981 0.481 0.29 0.3
0.38 0.136 0.911 0.231 0.556 0.798 0.496 0.407 0.237 0.474 0.676 0.356
0.757 0.954 0.217 0.165 0.948 0.746 0.986 0.501 0.216 0.638 0.398 0.863
0.462 0.924 0.889 0.448 0.325 0.922 0.895 0.331 0.491 0.626 0.207 0.133
0.68 0.304 0.126 0.835 0.233 0.485 0.217 0.405 0.44 0.124 0.71 0.332
0.546 0.58 0.151 0.447 0.104 0.206 0.257 0.053 0.716 0.804 0.67 0.789
0.804 0.473 0.008 0.318 0.033 0.381 0.634 0.407 0.659 0.62 0.497 0.689
0.83 0.384 0.67 0.911 0.101 0.668 0.355 0.579 0.111 0.446 0.596 0.814
0.318 0.355 0.07 0.542 0.017 0.21 0.327 0.599 0.059 0.252 0.951 0.56
0.367 0.813 0.074 0.964 0.079 0.68 0.446 0.019 0.7 0.903 0.918 0.74
0.22 0.241 0.656 0.283 0.625 0.209 0.154 0.862 0.254 0.151 0.323 0.789
0.393 0.023 0.668 0.55 0.408 0.54 0.207 0.064 0.844 0.323 0.216 0.688
0.273 0.71 0.542 0.32 0.277 0.535 0.621 0.014 0.272 0.235 0.959 0.067
0.027 0.585 0.001 0.853 0.189 0.687 0.059 0.284 0.419 0.995 0.151 0.391
0.184 0.741 0.752 0.956 0.646 0.84 0.619 0.993 0.37 0.499 0.491 0.318
0.782 0.724 0.748 0.552 0.485 0.667 0.206 0.813 0.511 0.128 0.936 0.33
0.937 0.484 0.157 0.878 0.834 0.133 0.809 0.977 0.567 0.366 0.964 0.535
0.678 0.64 0.076 0.866 0.211 0.853 0.619 0.103 0.433 0.667 0.73 0.136
0.519 0.612 0.184 0.044 0.448 0.233 0.885 0.38 0.172 0.804 0.106 0.724
0.107 0.619 0.554 0.548 0.812 0.587 0.577 0.417 0.962 0.774 0.364 0.485
0.881 0.533 0.714 0.52 0.963 0.718 0.651 0.375 0.889 0.239 0.148 0.715
0.551 0.768 0.073 0.599 0.671 0.947 0.059 0.453 0.356 0.271 0.156 0.096
0.975 0.454 0.594 0.605 0.689 0.151 0.823 0.286 0.107 0.031 0.59 0.801
0.847 0.291 0.516 0.977 0.883 0.169 0.848 0.954 0.371 0.632 0.313 0.397
0.944 0.937 0.051 0.193 0.221 0.446 0.327 0.456 0.619 0.924 0.326 0.848
0.496 0.515 0.668 0.703 0.942 0.712 0.533 0.656 0.691 0.669 0.407 0.42
0.659 0.933 1. 0.244 0.566 0.613 0.747 0.896 0.236 0.355 0.338 0.243
0.069 0.416 0.684 0.923 0.392 0.654 0.523 0.38 0.319 0.327 0.522 0.985
0.01 0.316 0.938 0.907]]
RECEIVED RESPONSE:
meta {
puid: "14k74obmqhus06jl6pai9hcg7r"
requestPath {
key: "classifier"
value: "gcr.io/dev-joel/deep-mnist-gpu:0.1"
}
}
data {
names: "class:0"
names: "class:1"
names: "class:2"
names: "class:3"
names: "class:4"
names: "class:5"
names: "class:6"
names: "class:7"
names: "class:8"
names: "class:9"
ndarray {
values {
list_value {
values {
number_value: 0.0025008211378008127
}
values {
number_value: 7.924897005295861e-08
}
values {
number_value: 0.057240355759859085
}
values {
number_value: 0.21792393922805786
}
values {
number_value: 6.878228759887861e-06
}
values {
number_value: 0.5588285326957703
}
values {
number_value: 0.0005614690016955137
}
values {
number_value: 0.0004520844086073339
}
values {
number_value: 0.161981999874115
}
values {
number_value: 0.0005038614035584033
}
}
}
}
}
Clean up¶
Make sure you delete the cluster once you have finished with it to avoid any ongoing charges.
[ ]:
!gcloud container clusters delete <YOUR_CLUSTER_NAME>