Troubleshooting Guide

If your Seldon Deployment does not seem to be running here are some tips to diagnose the issue.

My model does not seem to be running

Check whether the Seldon Deployment is running:

kubectl get sdep

If it exists, check it’s status, for a Seldon deployment called <name>:

kubectl get sdep <name> -o jsonpath='{.status}'

This might look like:

>kubectl get sdep

NAME      AGE
mymodel   1m

>kubectl get sdep mymodel -o jsonpath='{.status}'
map[predictorStatus:[map[name:mymodel-mymodel-7cd068f replicas:1 replicasAvailable:1]] state:Available]

If you have the jq tool installed you can get a nicer output with:

>kubectl get sdep mymodel -o json | jq .status
{
  "predictorStatus": [
    {
      "name": "mymodel-mymodel-7cd068f",
      "replicas": 1,
      "replicasAvailable": 1
    }
  ],
  "state": "Available"
}

For a model with invalid json/yaml an example is shown below:

>kubectl get sdep seldon-model -o json | jq .status
{
  "description": "Cannot find field: imagename in message k8s.io.api.core.v1.Container",
  "state": "Failed"
}

My Seldon Deployment remains in “creating” state

Check if the pods are running successfully.

I get 404s when calling the Ambassador endpoint

If your model is running and you are using Ambassador for ingress and are having problems check the diagnostics page of Ambassador. See here. You can then fnd out what path your model can be found under to ensure the URL you are using is correct.

If your ambassador isn’t running at all then check the pod logs with kubectl logs <pod_name>. Note that if ambassador is installed with cluster-wide scope then its rbac should also not be namespaced, otherwise a there will be a permissions error.

I get 500s when calling my model over the API

Check the logs of your running model pods.

My Seldon Deployment is not listed

Check the logs of the Seldon cluster-manager. This is the operator that handles the Seldon Deployment graphs sent to Kubernetes.

I have tried the above and I’m still confused