Guide to Benchmark Prediction Services

This guide will go through detailed steps to show how a Seldon prediction service can be benchmarked and will compare REST and gRPC variants. The service benchmarked will be the MNIST digit classifider demo. We will use the locust load testing framework which is easier to extend to create gRPC tests than Iago.

The steps taken will be

Create Kubernetes cluster on AWS

For our test we used AWS but other cloud providers can be used. You will need persistent storage to run Seldon. For this test we used GlusterFS.

Launch Seldon

   DATA_VOLUME="glusterfs": {"endpoints": "glusterfs-cluster","path": "gv0","readOnly": false}

Create the conf with make clean conf

   kubectl label nodes role=locust
   kubectl cordon

Download pretrained MNIST model

We follow the steps outlined in the MNIST digit classifier demo to setup a client and download a model for that client.

seldon-cli client --action setup --db-name ClientDB --client-name deep_mnist_client
cd kubernetes/conf/examples/tensorflow_deep_mnist
kubectl create -f load-model-tensorflow-deep-mnist.json

Wait until the model has finished downloading by checking the job with kubetctl get jobs

Start a REST MNIST prediction microservice

start-microservice --type prediction --client deep_mnist_client -p tensorflow-deep-mnist /seldon-data/seldon-models/tensorflow_deep_mnist/1/ rest 1.0

Run REST load test

   kubectl uncordon
  launch-locust-load-test --seldon-client deep_mnist_client --test-type js-predict

REST Results

locust REST results

Clean up the load test resources

   cd kubernetes/conf/microservices
kubectl delete -f microservice-tensorflow-deep-mnist.json
   cd kubernetes/conf/dev
kubectl delete -f locust-master.json
kubectl delete -f locust-slave.json
   kubectl cordon

Start the gRPC MNIST prediction microservice

start-microservice --type prediction --client deep_mnist_client -p tensorflow-deep-mnist /seldon-data/seldon-models/tensorflow_deep_mnist/1/ rpc 1.0

Run gRPC load test

   kubectl uncordon
  launch-locust-load-test --seldon-client deep_mnist_client --test-type grpc-predict

gRPC Results

locust gRPC results


On the average response time for REST is over 50% slower than for gRPC.

The percentiles from the two tests are as follows.

"Name",			"# requests",	"50%",	"66%",	"75%",	"80%",	"90%",	"95%",	"98%",	"99%",	"100%"
"grpc http://seldon-server:80" ,43521 14 16 18 19 25 36 67 110 5045
"GET /js/predict" ,42462 20 25 30 36 63 110 190 240 717

The advantage of gRPC increases as the percentiles increase except for the 100% percentile which suggests there was some delay at the start or outlier that needs further investigation.

The locust tests were done with 50 clients, each waiting between 900 and 1000ms between calls. Locust unlike Iago does not allow you to create a fixed request rate so some delays in the load testing framework itself between REST and gRPC implementations could effect the response times. However, we found it less easy to develop a gRPC test in Iago.