End to End Tests

Running the test suite


We use Kind, S2I and Kustomize.

Install python -m pip install grpcio-tools.


To get everything setup run:


Activate kind kubernetes config:

export KUBECONFIG="$(kind get kubeconfig-path)"


Then to run the tests and log output to a file:

make test > test.log


To view test logs in a separate terminal:

tail -f test.log

To also follow controller logs in a separate terminal:

export KUBECONFIG="$(kind get kubeconfig-path)"
kubectl logs -f -n seldon-system $(kubectl get pods -n seldon-system -l app=seldon -o jsonpath='{.items[0].metadata.name}')

Writing new tests

Invididual namespaces

Each test should run on its own separate Kubernetes namespace. This allows for a cleaner environment as well as enables some of them to get parallelised (see Sequential and parallel tests).

To make the creation and deletion of the namespace easier, you can use the namespace fixture. This fixture takes care of creating a namespace with a unique name and tearing it down at the end. To use it you just need to specify the namespace parameter as part of the arguments passed to your test function. The fixture will create the new namespace on the background and the namespace argument will take on the namespace name as value.

def test_foo(..., namespace, ...):
  print(f"the new namespace's name is {namespace}")

Sequential and parallel tests

We leverage pytest-xdist to speed up the test suite by parallelising the test execution. However, some of the tests may have side-effects which make them unsuitable to be executed alongside others. For example, the operator update tests change the cluster-wide Seldon operator which could affect the other tests running in parallel.

To differentiate between parallelisable tests and tests which are required to be run sequentially you can use the sequential mark. Marks in pytest allow to select subsets of tests.

To mark a test to run sequentially, you can do:

def test_foo(...):
  print("the scripts will run this test sequentially")

The integration test scripts will make sure that tests marked as sequential get run with a single worker.