RPC based services
The proto file for the Seldon gRPC services is shown below. It contains a single service Classifier for classification prediction calls that takes a ClassificationRequest and returns a ClassificationReply.
A ClassificationRequest has two parts
- meta : metadata associated with the request, presently an optional prediction-id the client wishes to associate with the request.
- data : A custom entry to hold the features needs for the prediction request defined by the user. This is optionally defined by the user. If not defined then a variable array of floats will be assumed as defined in DefaultCustomPredictRequest.
A ClassificationReply has three parts
- meta : metadata associated with the prediction
- puid : prediction unique id either supplied by client in request or created by Seldon
- modelName : the name of the model that satisfied the request
- variation : the AB test variation used in satisfying the request (will be “default” if a single variation).
- predictions : the predictions for each class
- custom : optional custom additional data that is defined by the user
Deploying a gRPC Prediction Service
The stages to deploy a gRPC service are shown below.
- (Optional) Create custom proto buffer file
- Build model and package microservice using gRPC
- (Optional) Inform Seldon of Java custom protocol buffers
- Launch gRPC microservice
- Test via REST or gRPC clients
Create Custom Proto Buffer
To hold the features required for a prediction request we will create the following proto buffer file:
Build model and package as microservice
To build the model we will use XGBoost and its wrapper provided in the pyseldon python library. The model is the same as shown in the Iris demo so won’t be discussed further here. We create python implementations of our protocol buffer and create a microservice wrapper to our built python pipeline model using the Microservice package as shown below. For this we need to create a custom data handler which will be called to extract a Pandas Dataframe from the RPC call data. The data will contain out custom IrisPredictRequest class which we need to unpack and translate into a Dataframe. The created Dataframe must match what is expected as input to your pipeline model.
The above model and resulting microservice has been packaged as a Docker container seldonio/iris_xgboost_rpc.
(Optional) Inform Seldon of Java custom protocol buffers
In this example case we have create a custom request protocol buffer. In order to process the gRPC calls for a client prediction endpoint correctly Seldon needs to know the custom request implementations. Assuming you have placed the iris.proto above on the Seldon shared volume at /seldon-data/rpc/proto/iris.proto you can run:
If you want to pass just a variable array of floats as the request data then this step is not required as this is the default.
Launch gRPC microservice
We can now launch our microservice using the script start-microservice
This will deploy our RPC microservice and inform Seldon of where it is running so requests can be sent to it.
Test via REST or gRPC clients
We can test our microservice via either the REST or gRPC endpoints of Seldon.
Test via JS REST API
To test the REST endpoint we can use the seldon-cli
This should produce a result like:
Test via gRPC client
To test the external gRPC endpoint we will need to create a gRPC client. To authenticate we will need to get an OAuth token using the Seldon REST endpoint. It needs to be passed in the header of the gRPC call with key “oauth_token”.
There is an example client for the Iris prediction in docker/examples/iris/xgboost_rpc/python/iris_rpc_client.py.
- Gets an Oauth token
- Calls the gRPC endpoint
It uses compiled python versions of the custom proto buffer and Seldon gRPC files located in the same folder.
You will need the Seldon host and ports for the HTTP and gRPC endpoints and the key and secret for your client. To get the oauth key and secret for client “test” run:
An example call for this client might be:
which should produce a similar result to the REST call above.