Distributed Tracing Templateยถ

Illustrate the configuration for allowing distributed tracing using Jaeger.

Dependenciesยถ

Test using Minikubeยถ

[4]:
!minikube start --memory 4096
๐Ÿ˜„  minikube v1.0.1 on linux (amd64)
๐Ÿคน  Downloading Kubernetes v1.14.1 images in the background ...
๐Ÿ”ฅ  Creating kvm2 VM (CPUs=2, Memory=4096MB, Disk=20000MB) ...
๐Ÿ“ถ  "minikube" IP address is 192.168.39.190
๐Ÿณ  Configuring Docker as the container runtime ...
๐Ÿณ  Version of container runtime is 18.06.3-ce
โŒ›  Waiting for image downloads to complete ...
โœจ  Preparing Kubernetes environment ...
๐Ÿšœ  Pulling images required by Kubernetes v1.14.1 ...
๐Ÿš€  Launching Kubernetes v1.14.1 using kubeadm ...
โŒ›  Waiting for pods: apiserver proxy etcd scheduler controller dns
๐Ÿ”‘  Configuring cluster permissions ...
๐Ÿค”  Verifying component health .....
๐Ÿ’—  kubectl is now configured to use "minikube"
๐Ÿ„  Done! Thank you for using minikube!
[5]:
!kubectl create clusterrolebinding kube-system-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
clusterrolebinding.rbac.authorization.k8s.io/kube-system-cluster-admin created
[6]:
!kubectl create namespace seldon
namespace/seldon created
[7]:
!kubectl config set-context $(kubectl config current-context) --namespace=seldon
Context "minikube" modified.
[8]:
!helm init
$HELM_HOME has been configured at /home/clive/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!
[9]:
!kubectl rollout status deploy/tiller-deploy -n kube-system
Waiting for deployment "tiller-deploy" rollout to finish: 0 of 1 updated replicas are available...
deployment "tiller-deploy" successfully rolled out

Install Jaegerยถ

We will use the Jaeger All-in-1 resource found at the Jaeger Kubernetes repo.

[10]:
!kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/all-in-one/jaeger-all-in-one-template.yml -n seldon
deployment.extensions/jaeger created
service/jaeger-query created
service/jaeger-collector created
service/jaeger-agent created
service/zipkin created

Start Jaeger UIยถ

minikube service jaeger-query -n seldon

Install Seldonยถ

[11]:
!helm install ../../../helm-charts/seldon-core-operator --name seldon-core --set usageMetrics.enabled=true --namespace seldon-system
NAME:   seldon-core
LAST DEPLOYED: Mon Jun 17 15:44:51 2019
NAMESPACE: seldon-system
STATUS: DEPLOYED

RESOURCES:
==> v1/ClusterRole
NAME                          AGE
seldon-operator-manager-role  0s

==> v1/ClusterRoleBinding
NAME                                 AGE
seldon-operator-manager-rolebinding  0s

==> v1/ConfigMap
NAME                     DATA  AGE
seldon-spartakus-config  3     1s

==> v1/Pod(related)
NAME                                         READY  STATUS             RESTARTS  AGE
seldon-operator-controller-manager-0         0/1    ContainerCreating  0         0s
seldon-spartakus-volunteer-5866b6df59-kpk4l  0/1    ContainerCreating  0         0s

==> v1/Secret
NAME                                   TYPE    DATA  AGE
seldon-operator-webhook-server-secret  Opaque  0     1s

==> v1/Service
NAME                                        TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)  AGE
seldon-operator-controller-manager-service  ClusterIP  10.110.128.191  <none>       443/TCP  0s

==> v1/ServiceAccount
NAME                              SECRETS  AGE
seldon-core-seldon-core-operator  1        1s
seldon-spartakus-volunteer        1        1s

==> v1/StatefulSet
NAME                                READY  AGE
seldon-operator-controller-manager  0/1    0s

==> v1beta1/ClusterRole
NAME                        AGE
seldon-spartakus-volunteer  0s

==> v1beta1/ClusterRoleBinding
NAME                        AGE
seldon-spartakus-volunteer  0s

==> v1beta1/CustomResourceDefinition
NAME                                         AGE
seldondeployments.machinelearning.seldon.io  0s

==> v1beta1/Deployment
NAME                        READY  UP-TO-DATE  AVAILABLE  AGE
seldon-spartakus-volunteer  0/1    1           0          0s


NOTES:
NOTES: TODO


[12]:
!kubectl rollout status statefulset.apps/seldon-operator-controller-manager -n seldon-system
Waiting for 1 pods to be ready...
partitioned roll out complete: 1 new pods have been updated...

Setup Ingressยถ

[13]:
!helm install stable/ambassador --name ambassador --set crds.keep=false
NAME:   ambassador
E0617 15:45:16.008304   10050 portforward.go:376] error copying from local connection to remote stream: read tcp4 127.0.0.1:42883->127.0.0.1:35738: read: connection reset by peer
LAST DEPLOYED: Mon Jun 17 15:45:15 2019
NAMESPACE: seldon
STATUS: DEPLOYED

RESOURCES:
==> v1/Deployment
NAME        READY  UP-TO-DATE  AVAILABLE  AGE
ambassador  0/3    3           0          1s

==> v1/Pod(related)
NAME                         READY  STATUS             RESTARTS  AGE
ambassador-778b689797-56npq  0/1    ContainerCreating  0         1s
ambassador-778b689797-phrjc  0/1    ContainerCreating  0         1s
ambassador-778b689797-qwx9z  0/1    ContainerCreating  0         1s

==> v1/Service
NAME               TYPE          CLUSTER-IP      EXTERNAL-IP  PORT(S)                     AGE
ambassador         LoadBalancer  10.100.134.88   <pending>    80:31764/TCP,443:32730/TCP  1s
ambassador-admins  ClusterIP     10.108.143.187  <none>       8877/TCP                    1s

==> v1/ServiceAccount
NAME        SECRETS  AGE
ambassador  1        1s

==> v1beta1/ClusterRole
NAME        AGE
ambassador  1s

==> v1beta1/ClusterRoleBinding
NAME        AGE
ambassador  1s

==> v1beta1/CustomResourceDefinition
NAME                                AGE
authservices.getambassador.io       1s
mappings.getambassador.io           1s
modules.getambassador.io            1s
ratelimitservices.getambassador.io  1s
tcpmappings.getambassador.io        1s
tlscontexts.getambassador.io        1s
tracingservices.getambassador.io    1s


NOTES:
Congratuations! You've successfully installed Ambassador.

For help, visit our Slack at https://d6e.co/slack or view the documentation online at https://www.getambassador.io.

To get the IP address of Ambassador, run the following commands:
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
     You can watch the status of by running 'kubectl get svc -w  --namespace seldon ambassador'

  On GKE/Azure:
  export SERVICE_IP=$(kubectl get svc --namespace seldon ambassador -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

  On AWS:
  export SERVICE_IP=$(kubectl get svc --namespace seldon ambassador -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

  echo http://$SERVICE_IP:

[14]:
!kubectl rollout status deployment.apps/ambassador
Waiting for deployment "ambassador" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "ambassador" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "ambassador" rollout to finish: 2 of 3 updated replicas are available...
deployment "ambassador" successfully rolled out

Create Jaeger ConfigMapยถ

[15]:
!pygmentize tracing-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: tracing-config
data:
  tracing.yml: |
    sampler:
      type: const
      param: 1
    local_agent:
      reporting_host: jaeger-agent
      reporting_port: 5775
    logging: true
[16]:
!kubectl apply -f tracing-configmap.yaml -n seldon
configmap/tracing-config created

Run Example REST Deploymentยถ

[17]:
!pygmentize deployment_rest.json
{
    "apiVersion": "machinelearning.seldon.io/v1alpha2",
    "kind": "SeldonDeployment",
    "metadata": {
        "labels": {
            "app": "seldon"
        },
        "name": "tracing-example",
        "namespace": "seldon"
    },
    "spec": {
        "name": "tracing-example",
        "oauth_key": "oauth-key",
        "oauth_secret": "oauth-secret",
        "predictors": [
            {
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "name": "model1",
                                "image": "seldonio/mock_classifier_rest:1.1",
                                "env": [
                                    {
                                        "name": "TRACING",
                                        "value": "1"
                                    },
                                    {
                                        "name": "JAEGER_CONFIG_PATH",
                                        "value": "/etc/tracing/config/tracing.yml"
                                    }
                                ],
                                "volumeMounts": [
                                    {
                                        "mountPath": "/etc/tracing/config",
                                        "name": "tracing-config"
                                    }
                                ]
                            }
                        ],
                        "terminationGracePeriodSeconds": 1,
                        "volumes": [
                            {
                                "name": "tracing-config",
                                "configMap": {
                                    "name": "tracing-config",
                                    "items": [
                                        {
                                            "key": "tracing.yml",
                                            "path":  "tracing.yml"
                                        }
                                    ]
                                }
                            }
                        ]
                    }
                }],
                "graph": {
                    "name": "model1",
                    "endpoint": { "type" : "REST" },
                    "type": "MODEL",
                    "children": [
                    ]
                },
                "name": "tracing",
                "replicas": 1,
                "svcOrchSpec" : {
                    "env": [
                        {
                            "name": "TRACING",
                            "value": "1"
                        },
                        {
                            "name": "JAEGER_AGENT_HOST",
                            "value": "jaeger-agent"
                        },
                        {
                            "name": "JAEGER_AGENT_PORT",
                            "value": "5775"
                        },
                        {
                            "name": "JAEGER_SAMPLER_TYPE",
                            "value": "const"
                        },
                        {
                            "name": "JAEGER_SAMPLER_PARAM",
                            "value": "1"
                        }
                    ]
                }
            }
        ]
    }
}
[18]:
!kubectl create -f deployment_rest.json
seldondeployment.machinelearning.seldon.io/tracing-example created
[1]:
!kubectl rollout status deployment/tracing-example-tracing-535f3a8
deployment "tracing-example-tracing-535f3a8" successfully rolled out
[2]:
!seldon-core-api-tester contract.json `minikube ip` `kubectl get svc ambassador -o jsonpath='{.spec.ports[0].nodePort}'` \
    tracing-example --namespace seldon -p
----------------------------------------
SENDING NEW REQUEST:

[[-2.146  0.92  -0.593]]
RECEIVED RESPONSE:
meta {
  puid: "o57qhtk6j6lql4rnl125j0fik3"
  requestPath {
    key: "model1"
    value: "seldonio/mock_classifier_rest:1.1"
  }
}
data {
  names: "proba"
  ndarray {
    values {
      list_value {
        values {
          number_value: 0.028664848186919598
        }
      }
    }
  }
}


Check the Jaeger UI. You should be able to find traces like below:

rest

[3]:
!kubectl delete -f deployment_rest.json
seldondeployment.machinelearning.seldon.io "tracing-example" deleted

Run Example GRPC Deploymentยถ

[4]:
!kubectl create -f deployment_grpc.json
seldondeployment.machinelearning.seldon.io/tracing-example created
[5]:
!kubectl rollout status deployment/tracing-example-tracing-d240ae0
Waiting for deployment "tracing-example-tracing-d240ae0" rollout to finish: 0 of 1 updated replicas are available...
deployment "tracing-example-tracing-d240ae0" successfully rolled out
[10]:
!seldon-core-api-tester contract.json `minikube ip` `kubectl get svc ambassador -o jsonpath='{.spec.ports[0].nodePort}'` \
    tracing-example --namespace seldon -p --grpc
----------------------------------------
SENDING NEW REQUEST:

[[ 0.643 -1.813  1.34 ]]
RECEIVED RESPONSE:
meta {
  puid: "i57ndibn2lo6coa98bsesvbs2t"
  requestPath {
    key: "model1"
    value: "seldonio/mock_classifier_grpc:1.1"
  }
}
data {
  names: "proba"
  ndarray {
    values {
      list_value {
        values {
          number_value: 0.05416670048123607
        }
      }
    }
  }
}


Check the Jaeger UI. You should be able to find traces like below:

grpc

[13]:
!kubectl delete -f deployment_grpc.json
seldondeployment.machinelearning.seldon.io "tracing-example" deleted