Custom LightGBM Prepackaged Model Server

In this notebook we create a new custom LIGHTGBM_SERVER prepackaged server with two versions: * A Seldon protocol LightGBM model server * A KfServing V2 protocol version using MLServer for running lightgbm models

The Seldon model server is in defined in lightgbmserver folder.

Prerequisites

  • A kubernetes cluster with kubectl configured

  • curl

Setup Seldon Core

Use the setup notebook to Setup Cluster to setup Seldon Core with an ingress - either Ambassador or Istio.

Then port-forward to that ingress on localhost:8003 in a separate terminal either with:

  • Ambassador: kubectl port-forward $(kubectl get pods -n seldon -l app.kubernetes.io/name=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8003:8080

  • Istio: kubectl port-forward $(kubectl get pods -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}') -n istio-system 8003:8080

[ ]:
!kubectl create namespace seldon
[ ]:
from IPython.core.magic import register_line_cell_magic


@register_line_cell_magic
def writetemplate(line, cell):
    with open(line, "w") as f:
        f.write(cell.format(**globals()))
[ ]:
VERSION = !cat ../../../version.txt
VERSION = VERSION[0]
VERSION

Training (can be skipped)

[ ]:
TRAIN_MODEL = False
if TRAIN_MODEL:
    import os

    import joblib
    import lightgbm as lgb
    from sklearn import datasets
    from sklearn.datasets import load_iris
    from sklearn.model_selection import train_test_split

    model_dir = "./artifacts"
    BST_FILE = "model.txt"

    iris = load_iris()
    y = iris["target"]
    X = iris["data"]
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1)
    dtrain = lgb.Dataset(X_train, label=y_train)

    params = {"objective": "multiclass", "metric": "softmax", "num_class": 3}
    lgb_model = lgb.train(params=params, train_set=dtrain)
    model_file = os.path.join(model_dir, BST_FILE)
    lgb_model.save_model(model_file)

Update Seldon Core with Custom Model

[ ]:
%%writetemplate values.yaml
predictor_servers:
  MLFLOW_SERVER:
    protocols:
      seldon:
        defaultImageVersion: "{VERSION}"
        image: seldonio/mlflowserver
  SKLEARN_SERVER:
    protocols:
      seldon:
        defaultImageVersion: "{VERSION}"
        image: seldonio/sklearnserver
      kfserving:
        defaultImageVersion: "0.3.2"
        image: seldonio/mlserver
  TENSORFLOW_SERVER:
    protocols:
      seldon:
        defaultImageVersion: "{VERSION}"
        image: seldonio/tfserving-proxy
      tensorflow:
        defaultImageVersion: 2.1.0
        image:  tensorflow/serving
  XGBOOST_SERVER:
    protocols:
      seldon:
        defaultImageVersion: "{VERSION}"
        image: seldonio/xgboostserver
      kfserving:
        defaultImageVersion: "0.3.2"
        image: seldonio/mlserver
  LIGHTGBM_SERVER:
    protocols:
      seldon:
        defaultImageVersion: "{VERSION}"
        image: seldonio/lighgbmserver
      kfserving:
        defaultImageVersion: "0.3.2"
        image: seldonio/mlserver
  TRITON_SERVER:
    protocols:
      kfserving:
        defaultImageVersion: "21.08-py3"
        image: nvcr.io/nvidia/tritonserver
  TEMPO_SERVER:
    protocols:
      kfserving:
        defaultImageVersion: "0.3.2"
        image: seldonio/mlserver

[ ]:
!helm upgrade seldon-core  \
    ../../../helm-charts/seldon-core-operator \
    --namespace seldon-system \
    --values values.yaml \
    --set istio.enabled=true

DeployLightGBM Model with Seldon Protocol

[ ]:
!cat model_seldon_v1.yaml

Wait for new webhook certificates to be loaded

[ ]:
import time

time.sleep(60)
[ ]:
!kubectl create -f model_seldon_v1.yaml -n seldon
[ ]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=iris -o jsonpath='{.items[0].metadata.name}' -n seldon) -n seldon
[ ]:
for i in range(60):
    state = !kubectl get sdep iris -n seldon -o jsonpath='{.status.state}'
    state = state[0]
    print(state)
    if state == "Available":
        break
    time.sleep(1)
assert state == "Available"
[ ]:
import json
X=!curl -s -d '{"data": {"ndarray":[[1.0, 2.0, 3.0, 4.0]]}}' \
   -X POST http://localhost:8003/seldon/seldon/iris/api/v1.0/predictions \
   -H "Content-Type: application/json"
d=json.loads(X[0])
print(d)
[ ]:
!kubectl delete -f model_seldon_v1.yaml

Deploy Model with KFserving Protocol

[ ]:
!cat model_seldon_v2.yaml
[ ]:
!kubectl create -f model_seldon_v2.yaml -n seldon
[ ]:
!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=iris -o jsonpath='{.items[0].metadata.name}' -n seldon) -n seldon
[ ]:
for i in range(60):
    state = !kubectl get sdep iris -n seldon -o jsonpath='{.status.state}'
    state = state[0]
    print(state)
    if state == "Available":
        break
    time.sleep(1)
assert state == "Available"
[ ]:
import json
X=!curl -s -d '{"inputs": [{"name": "predict", "shape": [1, 4], "datatype": "FP32", "data": [[1, 2, 3, 4]]}]}'\
   -X POST http://localhost:8003/seldon/seldon/iris/v2/models/infer \
   -H "Content-Type: application/json"
d=json.loads(X[0])
print(d)
[ ]:
!kubectl delete -f model_seldon_v2.yaml
[ ]: