Custom inference servers

Out of the box, Seldon offers support for several pre-packaged inference servers. However, there may be cases where it makes sense to rollout your own re-usable inference server. For example, you may need particular dependencies, specific versions, a custom process to download your model weights, etc.

To support these use cases, Seldon allows you to easily build your own inference servers, which can then be configured to be used as you would do the pre-packaged ones. That is, by using the implementation key and passing through model parameters (e.g. modelUri).

apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  name: nlp-model
spec:
  predictors:
    - graph:
        children: []
        implementation: CUSTOM_INFERENCE_SERVER
        modelUri: s3://our-custom-models/nlp-model
        name: model
      name: default
      replicas: 1

Building a new inference server

To build a new custom inference server you can follow the same instructions as the ones on how to wrap a model. The only difference is that the server will now receive a set of model parameters, like modelUri.

As inspiration, you can see how the SKLearn server and the other pre-packaged inference servers handle these as part of their __init__() method:

def __init__(self, model_uri: str = None,  method: str = "predict_proba"):
        super().__init__()
        self.model_uri = model_uri
        self.method = method
        self.ready = False
        self.load()

Adding a new inference server

The list of available inference servers in Seldon Core is maintained in the seldon-config configmap, which lives in the same namespace as your Seldon Core operator. In particular, the predictor_servers key holds the JSON config for each inference server.

The predictor_servers key will hold a JSON dictionary similar to the one below:

{
  "SKLEARN_SERVER": {
    "grpc": {
      "defaultImageVersion": "0.2",
      "image": "seldonio/sklearnserver_grpc"
    },
    "rest": {
      "defaultImageVersion": "0.2",
      "image": "seldonio/sklearnserver_rest"
    }
  }
}

Adding a new inference server is just a matter of adding a new key to the dict above. For example:

{
  "CUSTOM_INFERENCE_SERVER": {
    "grpc": {
      "defaultImageVersion": "1.0",
      "image": "org/custom-server-grpc"
    },
    "rest": {
      "defaultImageVersion": "1.0",
      "image": "org/custom-server-rest"
    }
  },
  "SKLEARN_SERVER": {
    "grpc": {
      "defaultImageVersion": "0.2",
      "image": "seldonio/sklearnserver_grpc"
    },
    "rest": {
      "defaultImageVersion": "0.2",
      "image": "seldonio/sklearnserver_rest"
    }
  }
}

Worked Example

A worked example to build a LighGBM Model server can be found here