This page was generated from examples/models/r_mnist/r_mnist.ipynb.

R MNIST Model

  • Wrap an R model (using the caret library) for use as a prediction microservice in seldon-core
  • Run locally on Docker to test
  • Deploy on seldon-core running on minikube

Dependencies

pip install seldon-core

Train locally

[11]:
!rm -f *ubyte && ./get_data.sh && Rscript train.R
--2019-04-24 16:43:55--  http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
Resolving yann.lecun.com (yann.lecun.com)... 216.165.22.6
Connecting to yann.lecun.com (yann.lecun.com)|216.165.22.6|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9912422 (9.5M) [application/x-gzip]
Saving to: ‘train-images-idx3-ubyte.gz’

train-images-idx3-u 100%[===================>]   9.45M  2.56MB/s    in 4.5s

2019-04-24 16:43:59 (2.10 MB/s) - ‘train-images-idx3-ubyte.gz’ saved [9912422/9912422]

--2019-04-24 16:43:59--  http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz
Resolving yann.lecun.com (yann.lecun.com)... 216.165.22.6
Connecting to yann.lecun.com (yann.lecun.com)|216.165.22.6|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 28881 (28K) [application/x-gzip]
Saving to: ‘train-labels-idx1-ubyte.gz’

train-labels-idx1-u 100%[===================>]  28.20K  --.-KB/s    in 0.08s

2019-04-24 16:44:00 (346 KB/s) - ‘train-labels-idx1-ubyte.gz’ saved [28881/28881]

--2019-04-24 16:44:00--  http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz
Resolving yann.lecun.com (yann.lecun.com)... 216.165.22.6
Connecting to yann.lecun.com (yann.lecun.com)|216.165.22.6|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1648877 (1.6M) [application/x-gzip]
Saving to: ‘t10k-images-idx3-ubyte.gz’

t10k-images-idx3-ub 100%[===================>]   1.57M  1.08MB/s    in 1.5s

2019-04-24 16:44:01 (1.08 MB/s) - ‘t10k-images-idx3-ubyte.gz’ saved [1648877/1648877]

--2019-04-24 16:44:01--  http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz
Resolving yann.lecun.com (yann.lecun.com)... 216.165.22.6
Connecting to yann.lecun.com (yann.lecun.com)|216.165.22.6|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4542 (4.4K) [application/x-gzip]
Saving to: ‘t10k-labels-idx1-ubyte.gz’

t10k-labels-idx1-ub 100%[===================>]   4.44K  --.-KB/s    in 0s

2019-04-24 16:44:01 (365 MB/s) - ‘t10k-labels-idx1-ubyte.gz’ saved [4542/4542]

Loading required package: lattice
Loading required package: ggplot2
Loading required package: foreach
Loading required package: iterators
Loading required package: parallel

Wrap model using s2i

[12]:
!s2i build . seldonio/seldon-core-s2i-r:0.1 r-mnist:0.1
---> Installing application source...
---> Installing dependencies ...
Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
trying URL 'https://cloud.r-project.org/src/contrib/pls_2.7-1.tar.gz'
Content type 'application/x-gzip' length 4390103 bytes (4.2 MB)
==================================================
downloaded 4.2 MB

* installing *source* package ‘pls’ ...
** package ‘pls’ successfully unpacked and MD5 sums checked
** R
** data
*** moving datasets to lazyload DB
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (pls)

The downloaded source packages are in
        ‘/tmp/RtmpSNT0ox/downloaded_packages’
Build completed successfully
[13]:
!docker run --name "mnist_predictor" -d --rm -p 5000:5000 r-mnist:0.1
5104ca6b9b9b9f8a778c46ae1c9d84a733ea63732dce128066306f7fcc49bb82

Send some random features that conform to the contract

[14]:
!seldon-core-tester contract.json 0.0.0.0 5000 -p
----------------------------------------
SENDING NEW REQUEST:

[[0.109 0.293 0.668 0.174 0.445 0.31  0.777 0.777 0.366 0.064 0.191 0.17
  0.254 0.693 0.425 0.566 0.443 0.931 0.575 0.524 0.978 0.797 0.42  0.928
  0.246 0.996 0.433 0.438 0.807 0.419 0.567 0.202 0.12  0.158 0.055 0.471
  0.347 0.491 0.988 0.559 0.482 0.922 0.566 0.182 0.19  0.977 0.933 0.541
  0.507 0.708 0.633 0.919 0.578 0.451 0.569 0.08  0.835 0.625 0.376 0.534
  0.18  0.123 0.212 0.186 0.254 0.369 0.93  0.276 0.565 0.738 0.951 0.222
  0.651 0.584 0.549 0.145 0.891 0.65  0.861 0.584 0.573 0.992 0.34  0.252
  0.122 0.998 0.066 0.883 0.606 0.651 0.764 0.591 0.314 0.048 0.269 0.843
  0.116 0.944 0.633 0.639 0.166 0.537 0.115 0.559 0.798 0.646 0.996 0.492
  0.178 0.77  0.107 0.109 0.302 0.559 0.497 0.382 0.35  0.764 0.744 0.75
  0.46  0.507 0.143 0.673 0.123 0.    0.569 0.442 0.485 0.923 0.018 0.16
  0.4   0.99  0.958 0.718 0.373 0.161 0.263 0.989 0.782 0.685 0.363 0.072
  0.177 0.757 0.383 0.628 0.067 0.876 0.78  0.479 0.627 0.732 0.625 0.307
  0.605 0.706 0.002 0.64  0.042 0.986 0.818 0.948 0.591 0.052 0.055 0.33
  0.32  0.37  0.298 0.703 0.077 0.469 0.612 0.514 0.769 0.252 0.862 0.16
  0.188 0.87  0.369 0.624 0.272 0.93  0.191 0.34  0.867 0.834 0.713 0.913
  0.477 0.791 0.416 0.69  0.939 0.046 0.572 0.265 0.478 0.439 0.614 0.655
  0.432 0.648 0.711 0.271 0.054 0.234 0.86  0.815 0.089 0.923 0.293 0.399
  0.069 0.751 0.773 0.588 0.928 0.846 0.397 0.721 0.81  0.225 0.48  0.415
  0.034 0.812 0.79  0.206 0.946 0.033 0.685 0.27  0.556 0.789 0.297 0.357
  0.854 0.219 0.859 0.096 0.978 0.374 0.631 0.669 0.522 0.249 0.613 0.259
  0.936 0.751 0.88  0.999 0.339 0.249 0.364 0.682 0.252 0.845 0.322 0.612
  0.071 0.812 0.351 0.61  0.617 0.088 0.609 0.718 0.035 0.656 0.624 0.189
  0.596 0.394 0.535 0.071 0.922 0.795 0.612 0.166 0.224 0.532 0.663 0.978
  0.882 0.78  0.744 0.087 0.911 0.44  0.334 0.49  0.29  0.903 0.831 0.062
  0.558 0.219 0.061 0.45  0.508 0.287 0.861 0.984 0.432 0.855 0.144 0.226
  0.655 0.762 0.999 0.714 0.435 0.4   0.034 0.772 0.777 0.845 0.421 0.103
  0.74  0.025 0.699 0.13  0.011 0.383 0.113 0.703 0.901 0.632 0.981 0.265
  0.398 0.785 0.314 0.907 0.152 0.477 0.059 0.423 0.792 0.023 0.949 0.901
  0.042 0.702 0.317 0.934 0.427 0.855 0.99  0.344 0.981 0.022 0.679 0.289
  0.821 0.296 0.314 0.759 0.846 0.692 0.448 0.576 0.608 0.087 0.053 0.783
  0.148 0.768 0.403 0.721 0.95  0.605 0.588 0.636 0.76  0.943 0.693 0.7
  0.272 0.422 0.593 0.49  0.103 0.425 0.376 0.342 0.002 0.576 0.851 0.385
  0.586 0.239 0.876 0.43  0.106 0.901 0.002 0.338 0.881 0.284 0.533 0.663
  0.375 0.957 0.954 0.278 0.508 0.262 0.293 0.529 0.623 0.92  0.428 0.843
  0.174 0.794 0.767 0.408 0.388 0.703 0.315 0.926 0.484 0.813 0.815 0.855
  0.783 0.354 0.436 0.62  0.371 0.543 0.532 0.577 0.712 0.389 0.12  0.297
  0.749 0.825 0.122 0.021 0.307 0.579 0.729 0.505 0.497 0.968 0.078 0.643
  0.858 1.    0.767 0.047 0.615 0.571 0.577 0.185 0.937 0.647 0.03  0.465
  0.231 0.477 0.594 0.96  0.448 0.924 0.192 0.676 0.82  0.454 0.001 0.498
  0.876 0.752 0.312 0.682 0.44  0.456 0.574 0.343 0.977 0.345 0.904 0.856
  0.433 0.448 0.3   0.917 0.662 0.63  0.358 0.134 0.784 0.314 0.876 0.564
  0.484 0.213 0.404 0.335 0.162 0.266 0.769 0.69  0.155 0.172 0.002 0.392
  0.137 0.7   0.367 0.874 0.963 0.194 0.397 0.021 0.221 0.632 0.467 0.669
  0.65  0.746 0.091 0.318 0.51  0.722 0.783 0.07  0.515 0.314 0.725 0.513
  0.576 0.961 0.826 0.846 0.395 0.073 0.936 0.983 0.105 0.295 0.237 0.812
  0.189 0.47  0.095 0.099 0.611 0.227 0.598 0.042 0.037 0.384 0.114 0.654
  0.835 0.429 0.161 0.261 0.068 0.204 0.937 0.803 0.031 0.653 0.644 0.072
  0.614 0.152 0.305 0.894 0.646 0.619 0.305 0.519 0.012 0.056 0.881 0.016
  0.921 0.974 0.856 0.872 0.906 0.933 0.372 0.51  0.188 0.921 0.438 0.291
  0.696 0.046 0.297 0.121 0.114 0.191 0.386 0.55  0.669 0.032 0.955 0.385
  0.941 0.513 0.946 0.061 0.631 0.574 0.619 0.892 0.135 0.091 0.53  0.252
  0.477 0.375 0.253 0.31  0.57  0.132 0.828 0.943 0.268 0.954 0.428 0.775
  0.96  0.69  0.974 0.338 0.397 0.41  0.02  0.817 0.401 0.928 0.962 0.925
  0.057 0.174 0.318 0.011 0.634 0.804 0.381 0.071 0.559 0.07  0.915 0.31
  0.777 0.194 0.47  0.305 0.827 0.232 0.904 0.689 0.447 0.986 0.474 0.167
  0.254 0.162 0.624 0.639 0.743 0.513 0.789 0.453 0.535 0.884 0.489 0.549
  0.482 0.369 0.039 0.358 0.022 0.707 0.415 0.013 0.538 0.235 0.051 0.932
  0.131 0.919 0.191 0.501 0.877 0.188 0.693 0.195 0.163 0.187 0.56  0.72
  0.221 0.526 0.963 0.237 0.351 0.43  0.251 0.083 0.348 0.622 0.613 0.929
  0.559 0.15  0.903 0.371 0.341 0.833 0.427 0.498 0.776 0.048 0.105 0.479
  0.97  0.839 0.481 0.978 0.14  0.039 0.241 0.294 0.269 0.849 0.645 0.458
  0.214 0.734 0.471 0.007 0.7   0.24  0.565 0.269 0.62  0.241 0.009 0.669
  0.173 0.53  0.462 0.058 0.254 0.164 0.611 0.706 0.515 0.501 0.985 0.561
  0.219 0.238 0.303 0.614 0.235 0.462 0.927 0.824 0.274 0.668 0.26  0.672
  0.75  0.401 0.06  0.356]]
RECEIVED RESPONSE:
data {
  names: "0"
  names: "1"
  names: "2"
  names: "3"
  names: "4"
  names: "5"
  names: "6"
  names: "7"
  names: "8"
  names: "9"
  ndarray {
    values {
      list_value {
        values {
          number_value: 0.1191
        }
        values {
          number_value: 0.0836
        }
        values {
          number_value: 0.1071
        }
        values {
          number_value: 0.0985
        }
        values {
          number_value: 0.1022
        }
        values {
          number_value: 0.0994
        }
        values {
          number_value: 0.1058
        }
        values {
          number_value: 0.0917
        }
        values {
          number_value: 0.0971
        }
        values {
          number_value: 0.0954
        }
      }
    }
  }
}


[15]:
!docker rm mnist_predictor --force
mnist_predictor

Test using Minikube

[ ]:
!minikube start --memory 4096
[ ]:
!kubectl create clusterrolebinding kube-system-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
[ ]:
!helm init
[ ]:
!kubectl rollout status deploy/tiller-deploy -n kube-system
[2]:
!helm install ../../../helm-charts/seldon-core-operator --name seldon-core --set usageMetrics.enabled=true   --namespace seldon-system
NAME:   seldon-core
LAST DEPLOYED: Wed Apr 24 16:36:50 2019
NAMESPACE: seldon-system
STATUS: DEPLOYED

RESOURCES:
==> v1/Secret
NAME                                   TYPE    DATA  AGE
seldon-operator-webhook-server-secret  Opaque  0     0s

==> v1beta1/CustomResourceDefinition
NAME                                         AGE
seldondeployments.machinelearning.seldon.io  0s

==> v1/ClusterRole
seldon-operator-manager-role  0s

==> v1/ClusterRoleBinding
NAME                                 AGE
seldon-operator-manager-rolebinding  0s

==> v1/Service
NAME                                        TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)  AGE
seldon-operator-controller-manager-service  ClusterIP  10.110.57.111  <none>       443/TCP  0s

==> v1/StatefulSet
NAME                                DESIRED  CURRENT  AGE
seldon-operator-controller-manager  1        1        0s

==> v1/Pod(related)
NAME                                  READY  STATUS             RESTARTS  AGE
seldon-operator-controller-manager-0  0/1    ContainerCreating  0         0s


NOTES:
NOTES: TODO


[4]:
!kubectl rollout status deploy/seldon-controller-manager -n seldon-system
partitioned roll out complete: 1 new pods have been updated...

Setup Ingress

Please note: There are reported gRPC issues with ambassador (see https://github.com/SeldonIO/seldon-core/issues/473).

[5]:
!helm install stable/ambassador --name ambassador --set crds.keep=false
NAME:   ambassador
LAST DEPLOYED: Wed Apr 24 16:37:34 2019
NAMESPACE: seldon
STATUS: DEPLOYED

RESOURCES:
==> v1/Deployment
NAME        DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
ambassador  3        3        3           0          0s

==> v1/Pod(related)
NAME                         READY  STATUS             RESTARTS  AGE
ambassador-5b89d44544-5rmss  0/1    ContainerCreating  0         0s
ambassador-5b89d44544-kzmp2  0/1    ContainerCreating  0         0s
ambassador-5b89d44544-qmf7z  0/1    ContainerCreating  0         0s

==> v1/ServiceAccount
NAME        SECRETS  AGE
ambassador  1        0s

==> v1beta1/ClusterRole
NAME        AGE
ambassador  0s

==> v1beta1/ClusterRoleBinding
NAME        AGE
ambassador  0s

==> v1/Service
NAME               TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)                     AGE
ambassador-admins  ClusterIP     10.98.164.148  <none>       8877/TCP                    0s
ambassador         LoadBalancer  10.111.173.75  <pending>    80:30659/TCP,443:31171/TCP  0s


NOTES:
Congratuations! You've successfully installed Ambassador.

For help, visit our Slack at https://d6e.co/slack or view the documentation online at https://www.getambassador.io.

To get the IP address of Ambassador, run the following commands:
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
     You can watch the status of by running 'kubectl get svc -w  --namespace seldon ambassador'

  On GKE/Azure:
  export SERVICE_IP=$(kubectl get svc --namespace seldon ambassador -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

  On AWS:
  export SERVICE_IP=$(kubectl get svc --namespace seldon ambassador -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

  echo http://$SERVICE_IP:

[6]:
!kubectl rollout status deployment.apps/ambassador
Waiting for deployment "ambassador" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "ambassador" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "ambassador" rollout to finish: 2 of 3 updated replicas are available...
deployment "ambassador" successfully rolled out
[16]:
!eval $(minikube docker-env) && s2i build . seldonio/seldon-core-s2i-r:0.1 r-mnist:0.1
---> Installing application source...
---> Installing dependencies ...
Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
trying URL 'https://cloud.r-project.org/src/contrib/pls_2.7-1.tar.gz'
Content type 'application/x-gzip' length 4390103 bytes (4.2 MB)
==================================================
downloaded 4.2 MB

* installing *source* package ‘pls’ ...
** package ‘pls’ successfully unpacked and MD5 sums checked
** R
** data
*** moving datasets to lazyload DB
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (pls)

The downloaded source packages are in
        ‘/tmp/RtmpDwBobW/downloaded_packages’
Build completed successfully
[17]:
!kubectl create -f r_mnist_deployment.json
seldondeployment.machinelearning.seldon.io/seldon-deployment-example created

Wait until ready (replicas == replicasAvailable)

[18]:
!kubectl rollout status deploy/r-mnist-deployment-r-mnist-predictor-d12e455
Waiting for deployment "r-mnist-deployment-r-mnist-predictor-d12e455" rollout to finish: 0 of 1 updated replicas are available...
deployment "r-mnist-deployment-r-mnist-predictor-d12e455" successfully rolled out
[1]:
!seldon-core-api-tester contract.json `minikube ip` `kubectl get svc ambassador -o jsonpath='{.spec.ports[0].nodePort}'` \
    seldon-deployment-example --namespace seldon -p
----------------------------------------
SENDING NEW REQUEST:

[[0.042 0.143 0.444 0.009 0.57  0.229 0.198 0.534 0.339 0.835 0.463 0.584
  0.228 0.242 0.581 0.451 0.497 0.252 0.791 0.457 0.817 0.167 0.194 0.088
  0.824 0.014 0.278 0.186 0.43  0.538 0.321 0.847 0.    0.325 0.244 0.208
  0.15  0.984 0.306 0.564 0.507 0.676 0.48  0.218 0.641 0.395 0.277 0.992
  0.136 0.525 0.063 0.092 0.45  0.635 0.935 0.331 0.082 0.336 0.425 0.918
  0.823 0.878 0.496 0.931 0.137 0.24  0.631 0.707 0.142 0.783 0.615 0.203
  0.511 0.97  0.313 0.68  0.61  0.4   0.408 0.41  0.853 0.361 0.988 0.378
  0.557 0.095 0.367 0.418 0.478 0.312 0.873 0.267 0.396 0.82  0.423 0.848
  0.109 0.101 0.326 0.254 0.552 0.817 0.981 0.729 0.822 0.246 0.331 0.61
  0.921 0.208 0.633 0.635 0.246 0.552 0.43  0.158 0.469 0.729 0.212 0.524
  0.223 0.18  0.829 0.101 0.765 0.589 0.795 0.464 0.101 0.934 0.136 0.005
  0.672 0.516 0.837 0.133 0.544 0.425 0.711 0.225 0.213 0.593 0.987 0.611
  0.842 0.601 0.745 0.546 0.177 0.794 0.534 0.901 0.76  0.018 0.367 0.43
  0.122 0.138 0.206 0.658 0.549 0.112 0.539 0.364 0.671 0.034 0.806 0.868
  0.144 0.023 0.618 0.559 0.267 0.002 0.865 0.415 0.948 0.214 0.476 0.254
  0.972 0.339 0.176 0.029 0.63  0.329 0.204 0.65  0.203 0.031 0.955 0.059
  0.321 0.328 0.283 0.877 0.01  0.726 0.232 0.116 0.173 0.253 0.956 0.218
  0.287 0.592 0.92  0.781 0.187 0.933 0.306 0.615 0.686 0.262 0.318 0.574
  0.51  0.255 0.069 0.548 0.548 0.153 0.835 0.82  0.669 0.616 0.766 0.864
  0.438 0.153 0.166 0.174 0.826 0.887 0.957 0.632 0.968 0.765 0.916 0.013
  0.348 0.53  0.234 0.093 0.439 0.039 0.516 0.836 0.88  0.215 0.508 0.51
  0.948 0.79  0.023 0.814 0.436 0.388 0.213 0.182 0.37  0.337 0.18  0.213
  0.581 0.39  0.787 0.605 0.781 0.757 0.008 0.534 0.857 0.099 0.313 0.443
  0.272 0.711 0.493 0.856 0.357 0.674 0.702 0.036 0.895 0.904 0.559 0.481
  0.792 0.278 0.734 0.531 0.692 0.437 0.331 0.558 0.196 0.998 0.023 0.862
  0.446 0.707 0.766 0.334 0.098 0.299 0.061 0.026 0.958 0.611 0.402 0.207
  0.69  0.445 0.676 0.041 0.95  0.149 0.034 0.096 0.211 0.197 0.222 0.12
  0.453 0.553 0.657 0.865 0.279 0.744 0.134 0.765 0.212 0.251 0.696 0.241
  0.757 0.111 0.074 0.002 0.361 0.775 0.714 0.533 0.789 0.692 0.653 0.206
  0.759 0.576 0.807 0.554 0.967 0.223 0.709 0.301 0.567 0.901 0.57  0.636
  0.739 0.533 0.347 0.514 0.207 0.266 0.459 0.102 0.034 0.428 0.261 0.594
  0.596 0.322 0.445 0.287 0.871 0.716 0.052 0.863 0.097 0.758 0.724 0.985
  0.247 0.82  0.043 0.457 0.936 0.476 0.721 0.746 0.675 0.632 0.176 0.63
  0.219 0.029 0.189 0.37  0.046 0.709 0.031 0.939 0.684 0.876 0.859 0.735
  0.958 0.615 0.095 0.387 0.057 0.199 0.604 0.163 0.762 0.574 0.657 0.492
  0.125 0.972 0.445 0.222 0.457 0.595 0.811 0.593 0.389 0.473 0.043 0.96
  0.989 0.661 0.116 0.548 0.591 0.871 0.735 0.108 0.727 0.837 0.793 0.114
  0.816 0.809 0.379 0.63  0.603 0.678 0.911 0.93  0.257 0.881 0.658 0.486
  0.328 0.723 0.395 0.738 0.693 0.176 0.177 0.022 0.758 0.595 0.756 0.966
  0.271 0.605 0.086 0.078 0.265 0.934 0.877 0.501 0.036 0.507 0.797 0.4
  0.456 0.238 0.629 0.422 0.185 0.278 0.203 0.067 0.867 0.876 0.822 0.628
  0.78  0.775 0.622 0.081 0.722 0.183 0.346 0.504 0.599 0.903 0.272 0.797
  0.156 0.545 0.752 0.842 0.918 0.339 0.212 0.859 0.57  0.983 0.355 0.463
  0.661 0.849 0.574 0.9   0.429 0.483 0.807 0.696 0.878 0.663 0.769 0.128
  0.826 0.218 0.61  0.1   0.639 0.734 0.492 0.344 0.463 0.286 0.809 0.258
  0.426 0.758 0.637 0.194 0.183 0.621 0.271 0.955 0.888 0.495 0.325 0.736
  0.595 0.363 0.455 0.936 0.695 0.399 0.366 0.726 0.28  0.724 0.309 0.079
  0.333 0.104 0.326 0.473 0.871 0.239 0.419 0.738 0.43  0.567 0.62  0.883
  0.894 0.67  0.124 0.43  0.419 0.896 0.703 0.288 0.827 0.874 0.377 0.667
  0.753 0.541 0.998 0.481 0.004 0.327 0.559 0.117 0.89  0.864 0.14  0.204
  0.138 0.111 0.503 0.968 0.763 0.613 0.989 0.801 0.773 0.619 0.985 0.599
  0.802 0.12  0.604 0.721 0.316 0.562 0.304 0.818 0.52  0.175 0.402 0.641
  0.728 0.157 0.072 0.982 0.197 0.299 0.752 0.479 0.872 0.178 0.343 0.075
  0.427 0.199 0.63  0.697 0.261 0.787 0.457 0.205 0.843 0.23  0.285 0.69
  0.965 0.264 0.37  0.122 0.634 0.644 0.158 0.007 0.218 0.1   0.376 0.506
  0.523 0.805 0.082 0.418 0.574 0.726 0.346 0.419 0.759 0.835 0.536 0.122
  0.365 0.581 0.523 0.41  0.069 0.616 0.885 0.685 0.165 0.192 0.982 0.254
  0.277 0.451 0.719 0.893 0.941 0.559 0.398 0.897 0.33  0.669 0.092 0.913
  0.085 0.172 0.416 0.501 0.662 0.851 0.99  0.552 0.13  0.738 0.48  0.665
  0.893 0.034 0.97  0.105 0.459 0.255 0.608 0.343 0.102 0.268 0.178 0.961
  0.27  0.96  0.545 0.538 0.098 0.184 0.986 0.433 0.045 0.736 0.674 0.01
  0.181 0.572 0.299 0.8   0.269 0.019 0.662 0.196 0.403 0.216 0.71  0.058
  0.488 0.927 0.786 0.18  0.485 0.717 0.285 0.384 0.143 0.701 0.323 0.65
  0.864 0.346 0.672 0.152 0.922 0.341 0.174 0.102 0.11  0.982 0.109 0.468
  0.503 0.94  0.022 0.206 0.951 0.48  0.377 0.09  0.657 0.71  0.552 0.201
  0.974 0.448 0.565 0.376]]
RECEIVED RESPONSE:
None

[ ]:
!minikube delete
[ ]: