Nodejs Tensorflow Example

  • Wrap a nodejs tensorflow model for use as a prediction microservice in seldon-core
  • Run locally on Docker to test

Dependencies

Train locally using npm commands

This model example takes an input of 10 different features and predicts a out for the same. For the training part it uses a random normally distributed input set of 100 rows i.e a data set of [100,10] and trains it for another random normally distributed data set of size [100,1]. For every prediction the model expects a dataset of dimension [r,10] where r is the num of input rows to be predicted.

[1]:
!make train && make clean_build
npm install
        ......] \ refresh-package-json:@tensorflow/tfjs-node: timing actiom
> @tensorflow/[email protected] install /home/clive/work/seldon-core/fork-seldon-core/examples/models/nodejs_tensorflow/node_modules/@tensorflow/tfjs-node
> node scripts/install.js

* Downloading libtensorflow
[                              ] Infinity/bps 0% 0.0s[                              ] 154364/bps 0% 128.0s[                              ] 1240158/bps 0% 15.9s[                              ] 3603148/bps 0% 5.4s[=                             ] 4762971/bps 1% 4.1s[=                             ] 5481686/bps 2% 3.5s[=                             ] 6214471/bps 3% 3.1s[=                             ] 6967373/bps 4% 2.7s[=                             ] 7131544/bps 4% 2.6s[==                            ] 6972763/bps 5% 2.7s[==                            ] 9257518/bps 7% 2.0s[===                           ] 11301908/bps 10% 1.6s[====                          ] 12855355/bps 13% 1.3s[=====                         ] 14330366/bps 15% 1.2s[======                        ] 15672547/bps 18% 1.0s[======                        ] 16246964/bps 20% 1.0s[======                        ] 5880849/bps 21% 2.6s[=======                       ] 6088044/bps 22% 2.5s[=======                       ] 6089241/bps 23% 2.5s[=======                       ] 6240503/bps 24% 2.4s[========                      ] 6469579/bps 25% 2.3s[========                      ] 6936442/bps 27% 2.1s[=========                     ] 7304197/bps 30% 1.9s[==========                    ] 7668265/bps 32% 1.7s[==========                    ] 8104748/bps 34% 1.6s[===========                   ] 8582095/bps 37% 1.4s[============                  ] 9004661/bps 39% 1.3s[=============                 ] 9577209/bps 43% 1.2s[==============                ] 9949360/bps 45% 1.1s[==============                ] 10078374/bps 47% 1.0s[===============               ] 10481581/bps 49% 0.9s[================              ] 10819946/bps 52% 0.9s[================              ] 11096666/bps 54% 0.8s[=================             ] 11347867/bps 56% 0.8s[==================            ] 11607371/bps 59% 0.7s[==================            ] 11842695/bps 61% 0.6s[===================           ] 12102357/bps 63% 0.6s[====================          ] 12291890/bps 65% 0.6s[====================          ] 12552382/bps 67% 0.5s[=====================         ] 12774990/bps 70% 0.5s[======================        ] 12976247/bps 72% 0.4s[======================        ] 13042719/bps 73% 0.4s[=======================       ] 13306582/bps 76% 0.4s[========================      ] 13591597/bps 79% 0.3s[========================      ] 13784480/bps 81% 0.3s[=========================     ] 13861252/bps 82% 0.2s[==========================    ] 14045382/bps 85% 0.2s[==========================    ] 14251651/bps 87% 0.2s[===========================   ] 14372631/bps 89% 0.1s[============================  ] 14648295/bps 92% 0.1s[============================  ] 14670301/bps 93% 0.1s[============================  ] 14564979/bps 94% 0.1s[============================= ] 14524195/bps 95% 0.1s[============================= ] 14460878/bps 96% 0.1s[============================= ] 14434699/bps 97% 0.0s[============================= ] 14384762/bps 97% 0.0s[==============================] 14313428/bps 98% 0.0s[==============================] 14325589/bps 99% 0.0s[==============================] 14319896/bps 100% 0.0s
* Building TensorFlow Node.js bindings

> [email protected] postinstall /home/clive/work/seldon-core/fork-seldon-core/examples/models/nodejs_tensorflow/node_modules/protobufjs
> node scripts/postinstall

npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN [email protected] No repository field.
npm WARN [email protected] No license field.

added 48 packages from 56 contributors and audited 61 packages in 9.829s
found 0 vulnerabilities

npm start

> [email protected] start /home/clive/work/seldon-core/fork-seldon-core/examples/models/nodejs_tensorflow
> node train.js

2019-05-10 06:56:33.691232: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
Epoch 0: loss = 1.1140578985214233
Epoch 1: loss = 1.0404443740844727
Epoch 2: loss = 1.0114623308181763
Epoch 3: loss = 0.994644284248352
Epoch 4: loss = 0.9810447692871094
Epoch 5: loss = 0.9564876556396484
Epoch 6: loss = 0.947548508644104
Epoch 7: loss = 0.9377892017364502
Epoch 8: loss = 0.9292038679122925
Epoch 9: loss = 0.9103612899780273
Epoch 10: loss = 0.9044468402862549
Epoch 11: loss = 0.8943670392036438
Epoch 12: loss = 0.8909915685653687
Epoch 13: loss = 0.8821757435798645
Epoch 14: loss = 0.8772059679031372
Epoch 15: loss = 0.8722608685493469
Epoch 16: loss = 0.870168149471283
Epoch 17: loss = 0.8628248572349548
Epoch 18: loss = 0.856920599937439
Epoch 19: loss = 0.8508269786834717
Epoch 20: loss = 0.8445506691932678
Epoch 21: loss = 0.8388644456863403
Epoch 22: loss = 0.8324810862541199
Epoch 23: loss = 0.8312572836875916
Epoch 24: loss = 0.8251888155937195
Epoch 25: loss = 0.8173127770423889
Epoch 26: loss = 0.8206360936164856
Epoch 27: loss = 0.825434684753418
Epoch 28: loss = 0.8106041550636292
Epoch 29: loss = 0.8014734387397766
Epoch 30: loss = 0.7964511513710022
Epoch 31: loss = 0.7898756265640259
Epoch 32: loss = 0.7860068082809448
Epoch 33: loss = 0.7900837659835815
Epoch 34: loss = 0.7788155674934387
Epoch 35: loss = 0.778168261051178
Epoch 36: loss = 0.774094820022583
Epoch 37: loss = 0.7649340033531189
Epoch 38: loss = 0.759834349155426
Epoch 39: loss = 0.7585961818695068
Epoch 40: loss = 0.7511364817619324
Epoch 41: loss = 0.7497982382774353
Epoch 42: loss = 0.7454034090042114
Epoch 43: loss = 0.7422577738761902
Epoch 44: loss = 0.7390987873077393
Epoch 45: loss = 0.7328671813011169
Epoch 46: loss = 0.7296737432479858
Epoch 47: loss = 0.7255033850669861
Epoch 48: loss = 0.7259540557861328
Epoch 49: loss = 0.7198896408081055
Epoch 50: loss = 0.7157299518585205
Epoch 51: loss = 0.7137295603752136
Epoch 52: loss = 0.7115896344184875
Epoch 53: loss = 0.7110546827316284
Epoch 54: loss = 0.7083038687705994
Epoch 55: loss = 0.7007032036781311
Epoch 56: loss = 0.6936700344085693
Epoch 57: loss = 0.693160891532898
Epoch 58: loss = 0.6876615881919861
Epoch 59: loss = 0.6804297566413879
Epoch 60: loss = 0.6776358485221863
Epoch 61: loss = 0.6728461980819702
Epoch 62: loss = 0.6687815189361572
Epoch 63: loss = 0.6673902869224548
Epoch 64: loss = 0.6670713424682617
Epoch 65: loss = 0.6624063849449158
Epoch 66: loss = 0.65739905834198
Epoch 67: loss = 0.6553966999053955
Epoch 68: loss = 0.6506110429763794
Epoch 69: loss = 0.6493582129478455
Epoch 70: loss = 0.6465271711349487
Epoch 71: loss = 0.6439094543457031
Epoch 72: loss = 0.6397424340248108
Epoch 73: loss = 0.6372050046920776
Epoch 74: loss = 0.6370261907577515
Epoch 75: loss = 0.6327844858169556
Epoch 76: loss = 0.6300538182258606
Epoch 77: loss = 0.6324681639671326
Epoch 78: loss = 0.6271001100540161
Epoch 79: loss = 0.6215335130691528
Epoch 80: loss = 0.6228755116462708
Epoch 81: loss = 0.6215202808380127
Epoch 82: loss = 0.6156829595565796
Epoch 83: loss = 0.6130117774009705
Epoch 84: loss = 0.6068021655082703
Epoch 85: loss = 0.6044408082962036
Epoch 86: loss = 0.6065412759780884
Epoch 87: loss = 0.6013280749320984
Epoch 88: loss = 0.5983843803405762
Epoch 89: loss = 0.5943615436553955
Epoch 90: loss = 0.5942131280899048
Epoch 91: loss = 0.5931912660598755
Epoch 92: loss = 0.5889885425567627
Epoch 93: loss = 0.5844202041625977
Epoch 94: loss = 0.5816053748130798
Epoch 95: loss = 0.5841403007507324
Epoch 96: loss = 0.5787111520767212
Epoch 97: loss = 0.5767002105712891
Epoch 98: loss = 0.5712048411369324
Epoch 99: loss = 0.5715557932853699
/home/clive/work/seldon-core/fork-seldon-core/examples/models/nodejs_tensorflow


   ╭───────────────────────────────────────────────────────────────╮
                                                                  
          New minor version of npm available! 6.4.16.9.0       
      Changelog: https://github.com/npm/cli/releases/tag/v6.9.0   
                  Run npm install -g npm to update!               
                                                                  
   ╰───────────────────────────────────────────────────────────────╯

rm -rf node_modules
rm -f package-lock.json

Training creates a model.json file and a weights.bin file which is utilized for prediction

Prediction using REST API on the docker container

[2]:
!s2i build . seldonio/seldon-core-s2i-nodejs:0.2-SNAPSHOT node-s2i-model-image:0.1
---> Installing application source...
---> Installing dependencies ...

> @tensorflow/[email protected] install /microservice/model/node_modules/@tensorflow/tfjs-node
> node scripts/install.js

* Downloading libtensorflow

* Building TensorFlow Node.js bindings

> [email protected] postinstall /microservice/model/node_modules/protobufjs
> node scripts/postinstall

npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN [email protected] No repository field.
npm WARN [email protected] No license field.

added 48 packages from 56 contributors and audited 61 packages in 8.775s
found 0 vulnerabilities

Build completed successfully
[3]:
!docker run --name "nodejs_tensorflow_predictor" -d --rm -p 5000:5000 node-s2i-model-image:0.1
6cc8e4bca5aff59b1a4f0613f4e61ac212bd513954f4c61c964c0cb237a35f34

Send some random features that conform to the contract

[4]:
!seldon-core-tester contract.json 0.0.0.0 5000 -p -t
----------------------------------------
SENDING NEW REQUEST:

[[5.742 7.559 0.876 8.28  0.631 5.414 0.392 0.822 8.55  9.548]]
RECEIVED RESPONSE:
data {
  names: "t:0"
  tensor {
    shape: 1
    shape: 1
    values: -1.732214331626892
  }
}


[5]:
!docker rm nodejs_tensorflow_predictor --force
nodejs_tensorflow_predictor

Prediction using GRPC API on the docker container

[6]:
!s2i build -E ./.s2i/environment_grpc . seldonio/seldon-core-s2i-nodejs:0.2-SNAPSHOT node-s2i-model-image:0.2
---> Installing application source...
---> Installing dependencies ...

> @tensorflow/[email protected] install /microservice/model/node_modules/@tensorflow/tfjs-node
> node scripts/install.js

* Downloading libtensorflow

* Building TensorFlow Node.js bindings

> [email protected] postinstall /microservice/model/node_modules/protobufjs
> node scripts/postinstall

npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN [email protected] No repository field.
npm WARN [email protected] No license field.

added 48 packages from 56 contributors and audited 61 packages in 7.096s
found 0 vulnerabilities

Build completed successfully
[7]:
!docker run --name "nodejs_tensorflow_predictor" -d --rm -p 5000:5000 node-s2i-model-image:0.2
3e824d60a31f688ed8c5e7cc95cb6e15ff72669faec8e219d6fee3a900794007

Send some random features that conform to the contract

[8]:
!seldon-core-tester contract.json 0.0.0.0 5000 -p -t --grpc
----------------------------------------
SENDING NEW REQUEST:

[[8.196e+00 9.259e+00 8.349e+00 9.000e-03 5.450e+00 5.363e+00 2.453e+00
  3.760e-01 4.719e+00 7.410e-01]]
RECEIVED RESPONSE:
data {
  names: "t:0"
  tensor {
    shape: 1
    shape: 1
    values: 3.598963499069214
  }
}


[9]:
!docker rm nodejs_tensorflow_predictor --force
nodejs_tensorflow_predictor

Test using Minikube

Due to a `minikube/s2i issue <https://github.com/SeldonIO/seldon-core/issues/253>`__ you will need `s2i >= 1.1.13 <https://github.com/openshift/source-to-image/releases/tag/v1.1.13>`__

[ ]:
!minikube start --memory 4096
[10]:
!kubectl create clusterrolebinding kube-system-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
clusterrolebinding.rbac.authorization.k8s.io/kube-system-cluster-admin created
[11]:
!helm init
$HELM_HOME has been configured at /home/clive/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!
[12]:
!kubectl rollout status deploy/tiller-deploy -n kube-system
Waiting for deployment "tiller-deploy" rollout to finish: 0 of 1 updated replicas are available...
deployment "tiller-deploy" successfully rolled out
[13]:
!helm install ../../../helm-charts/seldon-core-operator --name seldon-core --set usageMetrics.enabled=true   --namespace seldon-system
NAME:   seldon-core
LAST DEPLOYED: Fri May 10 07:06:05 2019
NAMESPACE: seldon-system
STATUS: DEPLOYED

RESOURCES:
==> v1/Secret
NAME                                   TYPE    DATA  AGE
seldon-operator-webhook-server-secret  Opaque  0     1s

==> v1beta1/CustomResourceDefinition
NAME                                         AGE
seldondeployments.machinelearning.seldon.io  1s

==> v1/ClusterRole
seldon-operator-manager-role  0s

==> v1/ClusterRoleBinding
NAME                                 AGE
seldon-operator-manager-rolebinding  0s

==> v1/Service
NAME                                        TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)  AGE
seldon-operator-controller-manager-service  ClusterIP  10.104.23.226  <none>       443/TCP  0s

==> v1/StatefulSet
NAME                                DESIRED  CURRENT  AGE
seldon-operator-controller-manager  1        1        0s

==> v1/Pod(related)
NAME                                  READY  STATUS             RESTARTS  AGE
seldon-operator-controller-manager-0  0/1    ContainerCreating  0         0s


NOTES:
NOTES: TODO


[14]:
!kubectl rollout status deploy/seldon-controller-manager -n seldon-system
Waiting for 1 pods to be ready...
partitioned roll out complete: 1 new pods have been updated...

Setup Ingress

Please note: There are reported gRPC issues with ambassador (see https://github.com/SeldonIO/seldon-core/issues/473).

[15]:
!helm install stable/ambassador --name ambassador --set crds.keep=false
NAME:   ambassador
LAST DEPLOYED: Fri May 10 07:06:20 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1beta1/ClusterRole
NAME        AGE
ambassador  0s

==> v1beta1/ClusterRoleBinding
NAME        AGE
ambassador  0s

==> v1/Service
NAME               TYPE          CLUSTER-IP      EXTERNAL-IP  PORT(S)                     AGE
ambassador-admins  ClusterIP     10.102.250.208  <none>       8877/TCP                    0s
ambassador         LoadBalancer  10.111.31.208   <pending>    80:31312/TCP,443:32348/TCP  0s

==> v1/Deployment
NAME        DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
ambassador  3        3        3           0          0s

==> v1/Pod(related)
NAME                         READY  STATUS             RESTARTS  AGE
ambassador-5b89d44544-97jqg  0/1    ContainerCreating  0         0s
ambassador-5b89d44544-lhd5t  0/1    ContainerCreating  0         0s
ambassador-5b89d44544-nzr6d  0/1    ContainerCreating  0         0s

==> v1/ServiceAccount
NAME        SECRETS  AGE
ambassador  1        0s


NOTES:
Congratuations! You've successfully installed Ambassador.

For help, visit our Slack at https://d6e.co/slack or view the documentation online at https://www.getambassador.io.

To get the IP address of Ambassador, run the following commands:
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
     You can watch the status of by running 'kubectl get svc -w  --namespace default ambassador'

  On GKE/Azure:
  export SERVICE_IP=$(kubectl get svc --namespace default ambassador -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

  On AWS:
  export SERVICE_IP=$(kubectl get svc --namespace default ambassador -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

  echo http://$SERVICE_IP:

[16]:
!kubectl rollout status deployment.apps/ambassador
Waiting for deployment "ambassador" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "ambassador" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "ambassador" rollout to finish: 2 of 3 updated replicas are available...
deployment "ambassador" successfully rolled out

Build image and test

[17]:
!eval $(minikube docker-env) && s2i build . seldonio/seldon-core-s2i-nodejs:0.2-SNAPSHOT node-s2i-model-image:0.1
---> Installing application source...
---> Installing dependencies ...

> @tensorflow/[email protected] install /microservice/model/node_modules/@tensorflow/tfjs-node
> node scripts/install.js

* Downloading libtensorflow

* Building TensorFlow Node.js bindings

> [email protected] postinstall /microservice/model/node_modules/protobufjs
> node scripts/postinstall

npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN [email protected] No repository field.
npm WARN [email protected] No license field.

added 48 packages from 56 contributors and audited 61 packages in 10.745s
found 0 vulnerabilities

Build completed successfully
[18]:
!kubectl create -f nodejs_tensorflow_deployment.json
seldondeployment.machinelearning.seldon.io/seldon-deployment-example created
[19]:
!kubectl rollout status deploy/seldon-cea8a97ce503f62508ad289c86fe0e27
Waiting for deployment "seldon-cea8a97ce503f62508ad289c86fe0e27" rollout to finish: 0 of 1 updated replicas are available...
deployment "seldon-cea8a97ce503f62508ad289c86fe0e27" successfully rolled out
[20]:
!seldon-core-api-tester contract.json `minikube ip` `kubectl get svc ambassador -o jsonpath='{.spec.ports[0].nodePort}'` \
    seldon-deployment-example --namespace default -p
----------------------------------------
SENDING NEW REQUEST:

[[8.804 4.565 8.703 3.414 6.527 2.084 4.716 0.56  0.08  6.324]]
RECEIVED RESPONSE:
meta {
  puid: "37h6oijvi0g2iu10hlvgcnojms"
  requestPath {
    key: "nodejs-tensorflow-predictor"
    value: "node-s2i-model-image:0.1"
  }
}
data {
  names: "t:0"
  ndarray {
    values {
      list_value {
        values {
          number_value: -0.05151659995317459
        }
      }
    }
  }
}


[ ]:
!minikube delete
[21]:
!make clean
rm -rf node_modules
rm -f package-lock.json
rm -f model.json
rm -f weights.bin
[ ]: