Benchmarking with Argo Worfklows & Vegeta

In this notebook we will dive into how you can run bench marking with batch processing with Argo Workflows, Seldon Core and Vegeta.

Dependencies:

  • Seldon core installed as per the docs with Istio as an ingress

  • Argo Workfklows installed in cluster (and argo CLI for commands)

Setup

Install Seldon Core

Use the notebook to set-up Seldon Core with Ambassador or Istio Ingress.

Note: If running with KIND you need to make sure do follow these steps as workaround to the /.../docker.sock known issue.

Install Argo Workflows

You can follow the instructions from the official Argo Workflows Documentation.

You also need to make sure that argo has permissions to create seldon deployments - for this you can just create a default-admin rolebinding as follows:

[2]:
!kubectl create rolebinding default-admin --clusterrole=admin --serviceaccount=default:default
rolebinding.rbac.authorization.k8s.io/default-admin created
[117]:
def get_results(results, print_results=True):
    final = {}
    if "average" in results:
        final["mean"] = results["average"] / 1e6
        if results.get("latencyDistribution", False):
            final["50th"] = results["latencyDistribution"][-5]["latency"] / 1e6
            final["90th"] = results["latencyDistribution"][-3]["latency"] / 1e6
            final["95th"] = results["latencyDistribution"][-2]["latency"] / 1e6
            final["99th"] = results["latencyDistribution"][-1]["latency"] / 1e6
        final["rate"] = results["rps"]
        final["errors"] = results["statusCodeDistribution"]
    else:
        final["mean"] = results["latencies"]["mean"] / 1e6
        final["50th"] = results["latencies"]["50th"] / 1e6
        final["90th"] = results["latencies"]["90th"] / 1e6
        final["95th"] = results["latencies"]["95th"] / 1e6
        final["99th"] = results["latencies"]["99th"] / 1e6
        final["rate"] = results["throughput"]
        final["errors"] = results["errors"]
    if print_results:
        print("Latencies:")
        print("\tmean:", final["mean"], "ms")
        print("\t50th:", final["50th"], "ms")
        print("\t90th:", final["90th"], "ms")
        print("\t95th:", final["95th"], "ms")
        print("\t99th:", final["99th"], "ms")
        print("")
        print("Rate:", str(final["rate"]) + "/s")
        print("Errors:", final["errors"])
    return final

Create Benchmark Argo Workflow

In order to create a benchmark, we created a simple argo workflow template so you can leverage the power of the helm charts.

Before we dive into the contents of the full helm chart, let’s first give it a try with some of the settings.

We will run a batch job that will set up a Seldon Deployment with 1 replicas and 4 cpus (with 100 max workers) to send requests.

[2]:
!helm template seldon-benchmark-workflow helm-charts/seldon-benchmark-workflow/ \
    --set workflow.name=seldon-benchmark-process \
    --set seldonDeployment.name=sklearn \
    --set seldonDeployment.replicas=1 \
    --set seldonDeployment.serverWorkers=1 \
    --set seldonDeployment.serverThreads=10 \
    --set seldonDeployment.modelUri="gs://seldon-models/sklearn/iris" \
    --set seldonDeployment.server="SKLEARN_SERVER" \
    --set seldonDeployment.apiType=rest \
    --set benchmark.cpus=4 \
    --set benchmark.maxWorkers=100 \
    --set benchmark.duration=30s \
    --set benchmark.rate=0 \
    --set benchmark.data='\{"data": {"ndarray": [[0\,1\,2\,3]]\}\}' \
    | argo submit -
Name:                seldon-benchmark-process
Namespace:           default
ServiceAccount:      default
Status:              Pending
Created:             Fri Aug 07 18:09:40 +0100 (now)
[3]:
!argo list
NAME                       STATUS      AGE   DURATION   PRIORITY
seldon-benchmark-process   Succeeded   2m    1m         0
[4]:
!argo get seldon-benchmark-process
Name:                seldon-benchmark-process
Namespace:           default
ServiceAccount:      default
Status:              Succeeded
Created:             Fri Aug 07 18:09:40 +0100 (2 minutes ago)
Started:             Fri Aug 07 18:09:40 +0100 (2 minutes ago)
Finished:            Fri Aug 07 18:11:09 +0100 (51 seconds ago)
Duration:            1 minute 29 seconds

STEP                                                             PODNAME                              DURATION  MESSAGE
  seldon-benchmark-process (seldon-benchmark-process)
 ├--- create-seldon-resource (create-seldon-resource-template)  seldon-benchmark-process-3980407503  2s
 ├--- wait-seldon-resource (wait-seldon-resource-template)      seldon-benchmark-process-2136965893  49s
 └--- run-benchmark (run-benchmark-template)                    seldon-benchmark-process-780051119   32s
[5]:
!argo logs -w seldon-benchmark-process
create-seldon-resource:    time="2020-08-07T17:09:41.804Z" level=info msg="Starting Workflow Executor" version=v2.9.3
create-seldon-resource:    time="2020-08-07T17:09:41.809Z" level=info msg="Creating a docker executor"
create-seldon-resource:    time="2020-08-07T17:09:41.809Z" level=info msg="Executor (version: v2.9.3, build_date: 2020-07-18T19:11:19Z) initialized (pod: default/seldon-benchmark-process-3980407503) with template:\n{\"name\":\"create-seldon-resource-template\",\"arguments\":{},\"inputs\":{},\"outputs\":{},\"metadata\":{},\"resource\":{\"action\":\"create\",\"manifest\":\"apiVersion: machinelearning.seldon.io/v1\\nkind: SeldonDeployment\\nmetadata:\\n  name: \\\"sklearn\\\"\\n  namespace: default\\n  ownerReferences:\\n  - apiVersion: argoproj.io/v1alpha1\\n    blockOwnerDeletion: true\\n    kind: Workflow\\n    name: \\\"seldon-benchmark-process\\\"\\n    uid: \\\"e0364966-b2c1-4ee7-a7cf-421952ba3c7a\\\"\\nspec:\\n  annotations:\\n    seldon.io/executor: \\\"false\\\"\\n  name: \\\"sklearn\\\"\\n  transport: rest\\n  predictors:\\n    - componentSpecs:\\n      - spec:\\n        containers:\\n        - name: classifier\\n          env:\\n          - name: GUNICORN_THREADS\\n            value: 10\\n          - name: GUNICORN_WORKERS\\n            value: 1\\n          resources:\\n            requests:\\n              cpu: 50m\\n              memory: 100Mi\\n            limits:\\n              cpu: 50m\\n              memory: 1000Mi\\n      graph:\\n        children: []\\n        implementation: SKLEARN_SERVER\\n        modelUri: gs://seldon-models/sklearn/iris\\n        name: classifier\\n      name: default\\n      replicas: 1\\n\"}}"
create-seldon-resource:    time="2020-08-07T17:09:41.809Z" level=info msg="Loading manifest to /tmp/manifest.yaml"
create-seldon-resource:    time="2020-08-07T17:09:41.810Z" level=info msg="kubectl create -f /tmp/manifest.yaml -o json"
create-seldon-resource:    time="2020-08-07T17:09:42.456Z" level=info msg=default/SeldonDeployment.machinelearning.seldon.io/sklearn
create-seldon-resource:    time="2020-08-07T17:09:42.457Z" level=info msg="No output parameters"
wait-seldon-resource:        Waiting for deployment "sklearn-default-0-classifier" rollout to finish: 0 of 1 updated replicas are available...
wait-seldon-resource:        deployment "sklearn-default-0-classifier" successfully rolled out
run-benchmark:     {"latencies":{"total":3011298973622,"mean":339033885,"50th":272840630,"90th":339539236,"95th":368299307,"99th":4982426813,"max":5597505277,"min":206244298},"bytes_in":{"total":3081764,"mean":346.9673496960144},"bytes_out":{"total":301988,"mean":34},"earliest":"2020-08-07T17:10:37.117884325Z","latest":"2020-08-07T17:11:07.118729145Z","end":"2020-08-07T17:11:07.366654843Z","duration":30000844820,"wait":247925698,"requests":8882,"rate":296.05832946673667,"throughput":293.63176909007353,"success":1,"status_codes":{"200":8882},"errors":[]}
[6]:
import json
wf_logs = !argo logs -w seldon-benchmark-process
wf_bench = wf_logs[-1]
wf_json_str = wf_bench[24:]
results = json.loads(wf_json_str)

print("Latencies:")
print("\tmean:", results["latencies"]["mean"] / 1e6, "ms")
print("\t50th:", results["latencies"]["50th"] / 1e6, "ms")
print("\t90th:", results["latencies"]["90th"] / 1e6, "ms")
print("\t95th:", results["latencies"]["95th"] / 1e6, "ms")
print("\t99th:", results["latencies"]["99th"] / 1e6, "ms")
print("")
print("Throughput:", str(results["throughput"]) + "/s")
print("Errors:", len(results["errors"]) > 0)
Latencies:
        mean: 339.033885 ms
        50th: 272.84063 ms
        90th: 339.539236 ms
        95th: 368.299307 ms
        99th: 4982.426813 ms

Throughput: 293.63176909007353/s
Errors: False
[7]:
!argo delete seldon-benchmark-process
Workflow 'seldon-benchmark-process' deleted

Create GRPC benchmark with GHZ and Argo Workflows

[24]:
!helm template seldon-benchmark-workflow helm-charts/seldon-benchmark-workflow/ \
    --set workflow.name=seldon-benchmark-process \
    --set seldonDeployment.name=sklearn \
    --set seldonDeployment.replicas=1 \
    --set seldonDeployment.serverWorkers=1 \
    --set seldonDeployment.serverThreads=10 \
    --set seldonDeployment.modelUri="gs://seldon-models/sklearn/iris" \
    --set seldonDeployment.server="SKLEARN_SERVER" \
    --set seldonDeployment.apiType=grpc \
    --set benchmark.cpus=4 \
    --set benchmark.maxWorkers=100 \
    --set benchmark.duration="120s" \
    --set benchmark.rate=0 \
    --set benchmark.data='\{"data": {"ndarray": [[0\,1\,2\,3]]\}\}' \
    | argo submit -
Name:                seldon-benchmark-process
Namespace:           default
ServiceAccount:      default
Status:              Pending
Created:             Fri Aug 07 18:22:38 +0100 (now)
[25]:
!argo list
NAME                       STATUS      AGE   DURATION   PRIORITY
seldon-benchmark-process   Succeeded   4m    2m         0
[26]:
!argo get seldon-benchmark-process
Name:                seldon-benchmark-process
Namespace:           default
ServiceAccount:      default
Status:              Succeeded
Created:             Fri Aug 07 18:22:38 +0100 (4 minutes ago)
Started:             Fri Aug 07 18:22:38 +0100 (4 minutes ago)
Finished:            Fri Aug 07 18:25:11 +0100 (1 minute ago)
Duration:            2 minutes 33 seconds

STEP                                                             PODNAME                              DURATION  MESSAGE
  seldon-benchmark-process (seldon-benchmark-process)
 ├--- create-seldon-resource (create-seldon-resource-template)  seldon-benchmark-process-3980407503  2s
 ├--- wait-seldon-resource (wait-seldon-resource-template)      seldon-benchmark-process-2136965893  26s
 └--- run-benchmark (run-benchmark-template)                    seldon-benchmark-process-780051119   2m
[27]:
!argo logs -w seldon-benchmark-process
create-seldon-resource:    time="2020-08-07T17:22:39.446Z" level=info msg="Starting Workflow Executor" version=v2.9.3
create-seldon-resource:    time="2020-08-07T17:22:39.450Z" level=info msg="Creating a docker executor"
create-seldon-resource:    time="2020-08-07T17:22:39.450Z" level=info msg="Executor (version: v2.9.3, build_date: 2020-07-18T19:11:19Z) initialized (pod: default/seldon-benchmark-process-3980407503) with template:\n{\"name\":\"create-seldon-resource-template\",\"arguments\":{},\"inputs\":{},\"outputs\":{},\"metadata\":{},\"resource\":{\"action\":\"create\",\"manifest\":\"apiVersion: machinelearning.seldon.io/v1\\nkind: SeldonDeployment\\nmetadata:\\n  name: \\\"sklearn\\\"\\n  namespace: default\\n  ownerReferences:\\n  - apiVersion: argoproj.io/v1alpha1\\n    blockOwnerDeletion: true\\n    kind: Workflow\\n    name: \\\"seldon-benchmark-process\\\"\\n    uid: \\\"e472d69d-44ed-4a45-86b3-d4b64146002b\\\"\\nspec:\\n  name: \\\"sklearn\\\"\\n  transport: grpc\\n  predictors:\\n    - componentSpecs:\\n      - spec:\\n        containers:\\n        - name: classifier\\n          env:\\n          - name: GUNICORN_THREADS\\n            value: 10\\n          - name: GUNICORN_WORKERS\\n            value: 1\\n      graph:\\n        children: []\\n        implementation: SKLEARN_SERVER\\n        modelUri: gs://seldon-models/sklearn/iris\\n        name: classifier\\n      name: default\\n      replicas: 1\\n\"}}"
create-seldon-resource:    time="2020-08-07T17:22:39.450Z" level=info msg="Loading manifest to /tmp/manifest.yaml"
create-seldon-resource:    time="2020-08-07T17:22:39.450Z" level=info msg="kubectl create -f /tmp/manifest.yaml -o json"
create-seldon-resource:    time="2020-08-07T17:22:40.060Z" level=info msg=default/SeldonDeployment.machinelearning.seldon.io/sklearn
create-seldon-resource:    time="2020-08-07T17:22:40.060Z" level=info msg="No output parameters"
wait-seldon-resource:        Waiting for deployment "sklearn-default-0-classifier" rollout to finish: 0 of 1 updated replicas are available...
wait-seldon-resource:        deployment "sklearn-default-0-classifier" successfully rolled out
run-benchmark:     {"date":"2020-08-07T17:25:09Z","endReason":"timeout","options":{"host":"istio-ingressgateway.istio-system.svc.cluster.local:80","proto":"/proto/prediction.proto","import-paths":["/proto","."],"call":"seldon.protos.Seldon/Predict","insecure":true,"total":2147483647,"concurrency":50,"connections":1,"duration":120000000000,"timeout":20000000000,"dial-timeout":10000000000,"data":{"data":{"ndarray":[[0,1,2,3]]}},"binary":false,"metadata":{"namespace":"default","seldon":"sklearn"},"CPUs":4},"count":88874,"total":120001033613,"average":67376309,"fastest":21863600,"slowest":148816057,"rps":740.6102874631579,"errorDistribution":{"rpc error: code = Unavailable desc = transport is closing":50},"statusCodeDistribution":{"OK":88824,"Unavailable":50},"latencyDistribution":[{"percentage":10,"latency":54583101},{"percentage":25,"latency":59326600},{"percentage":50,"latency":65257398},{"percentage":75,"latency":73167799},{"percentage":90,"latency":82939600},{"percentage":95,"latency":89598800},{"percentage":99,"latency":101463001}]}
[28]:
import json
wf_logs = !argo logs -w seldon-benchmark-process
wf_bench = wf_logs[-1]
wf_json_str = wf_bench[24:]
results = json.loads(wf_json_str)

print("Latencies:")
print("\tmean:", results["average"] / 1e6, "ms")
print("\t50th:", results["latencyDistribution"][-5]["latency"] / 1e6, "ms")
print("\t90th:", results["latencyDistribution"][-3]["latency"] / 1e6, "ms")
print("\t95th:", results["latencyDistribution"][-2]["latency"] / 1e6, "ms")
print("\t99th:", results["latencyDistribution"][-1]["latency"] / 1e6, "ms")
print("")
print("Rate:", str(results["rps"]) + "/s")
print("Errors:", results["statusCodeDistribution"].get("Unavailable", 0) > 0)
print("Errors:", results["statusCodeDistribution"])
Latencies:
        mean: 67.376309 ms
        50th: 65.257398 ms
        90th: 82.9396 ms
        95th: 89.5988 ms
        99th: 101.463001 ms

Rate: 740.6102874631579/s
Errors: True
Errors: {'OK': 88824, 'Unavailable': 50}
[22]:
!argo delete seldon-benchmark-process
Workflow 'seldon-benchmark-process' deleted

Run a set of tests

We can now leverage the helm charts we created above to run a grid search on a set of parameters.

[ ]:
import itertools as it
import json
import time

grid_opts = {
    "A-replicas": [1, 3],
    "B-serverWorkers": [1, 4],
    "C-serverThreads": [50, 200],
    "D-apiType": ["rest", "grpc"],
    "E-cpus": [1, 4],
    "F-maxWorkers": [100, 300],
    "G-useEngine": ["true", "false"],
}

allNames = sorted(grid_opts)
combinations = it.product(*(grid_opts[Name] for Name in allNames))
all_results = []
for curr_values in combinations:
    print("VALUES:", curr_values)
    replicas, server_workers, server_threads, api_type, cpus, max_wokers, use_engine = curr_values

    # For some reason python vars don't work with multiline helm charts
    %env REPLICAS=$replicas
    %env SERVER_WORKERS=$server_workers
    %env SERVER_THREADS=$server_threads
    %env API_TYPE=$api_type
    %env CPUS=$cpus
    %env MAX_WORKERS=$max_wokers
    %env USE_ENGINE=$use_engine

    !helm template seldon-benchmark-workflow helm-charts/seldon-benchmark-workflow/ \
        --set workflow.name=seldon-benchmark-process \
        --set seldonDeployment.name=sklearn \
        --set seldonDeployment.replicas=$REPLICAS \
        --set seldonDeployment.serverWorkers=$SERVER_WORKERS \
        --set seldonDeployment.serverThreads=$SERVER_THREADS \
        --set seldonDeployment.apiType=$API_TYPE \
        --set seldonDeployment.useEngine=\"$USE_ENGINE\" \
        --set benchmark.cpus=$CPUS \
        --set benchmark.maxWorkers=$MAX_WORKERS \
        --set benchmark.duration=120s \
        --set benchmark.rate=0 \
        --set benchmark.data='\{"data": {"ndarray": [[0\,1\,2\,3]]\}\}' \
        | argo submit --wait -

    !argo wait seldon-benchmark-process

    wf_logs = !argo logs -w seldon-benchmark-process
    wf_bench = wf_logs[-1]
    wf_json_str = wf_bench[24:]
    results = json.loads(wf_json_str)

    result = get_results(results)
    result["replicas"] = replicas
    result["server_workers"] = server_workers
    result["server_threads"] = server_threads
    result["apiType"] = api_type
    result["cpus"] = cpus
    result["max_wokers"] = max_wokers
    result["use_engine"] = use_engine
    all_results.append(result)

    !argo delete seldon-benchmark-process
    time.sleep(1)
    print("\n\n")

Deeper Analysis

Now that we have all the parameters, we can do a deeper analysis

[186]:
import pandas as pd
df = pd.DataFrame.from_dict(results)
df.head()
[186]:
replicas server_workers server_threads apiType cpus max_wokers use_engine mean 50th 90th 95th 99th rate errors
0 1 1 50 rest 1 200 true 489.269344 455.617128 612.294382 672.510108 832.322767 407.879172 []
1 1 1 50 rest 1 200 false 529.767457 514.151876 591.278115 621.463805 749.348556 376.649458 []
2 1 1 50 rest 4 200 true 547.618426 526.472215 661.947413 720.039676 863.596098 364.363839 []
3 1 1 50 rest 4 200 false 593.880113 602.945695 737.993290 770.777543 1003.510371 336.075411 []
4 1 1 50 grpc 1 200 true 95.322943 97.896699 117.221999 125.852400 141.615501 523.628160 {'OK': 62790, 'Unavailable': 50}

GRPC as expected outperforms REST

[189]:
df.sort_values("rate", ascending=False)
[189]:
replicas server_workers server_threads apiType cpus max_wokers use_engine mean 50th 90th 95th 99th rate errors
60 3 4 200 grpc 1 200 true 31.389861 23.769589 71.583795 78.881398 91.312797 1586.593680 {'OK': 190361, 'Unavailable': 48}
52 3 4 50 grpc 1 200 true 31.398451 26.313000 64.841515 73.035800 88.744198 1586.555365 {'OK': 190333, 'Unavailable': 71}
45 3 1 200 grpc 1 200 false 32.191240 30.448302 60.616301 68.724406 91.484308 1547.003054 {'OK': 185606, 'Unavailable': 49}
61 3 4 200 grpc 1 200 false 32.727674 28.483400 63.750796 72.597310 90.693812 1521.590875 {'OK': 182555, 'Unavailable': 49}
55 3 4 50 grpc 4 200 false 33.629848 29.610701 67.065895 77.773100 97.296599 1479.320474 {'OK': 177471, 'Unavailable': 50}
... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
10 1 1 200 rest 4 200 true 571.452398 556.699256 693.093315 751.197598 1024.233714 348.889260 []
11 1 1 200 rest 4 200 false 587.900216 556.869872 723.744376 774.244702 939.994423 339.396160 []
3 1 1 50 rest 4 200 false 593.880113 602.945695 737.993290 770.777543 1003.510371 336.075411 []
8 1 1 200 rest 1 200 true 633.043624 617.853285 741.229073 776.560578 1846.623159 314.908167 []
9 1 1 200 rest 1 200 false 641.530606 653.922529 802.558303 847.414484 1570.484029 310.839312 []

64 rows × 14 columns

Deeper dive REST

As expected replicas has the biggest impact. It seems the parameters on the benchmark worker don’t seem to affect throughput.

[190]:
df[df["apiType"]=="rest"].sort_values("rate", ascending=False)
[190]:
replicas server_workers server_threads apiType cpus max_wokers use_engine mean 50th 90th 95th 99th rate errors
41 3 1 200 rest 1 200 false 201.167546 8.844305 629.250888 690.807158 809.635194 992.298652 []
48 3 4 50 rest 1 200 true 208.429576 11.377699 655.466848 685.265506 758.664504 957.846772 []
51 3 4 50 rest 4 200 false 211.228526 13.592301 641.484819 675.713639 795.682869 945.090980 []
59 3 4 200 rest 4 200 false 214.358834 13.573121 670.449768 690.048496 722.537613 930.694079 []
57 3 4 200 rest 1 200 false 216.646320 9.336961 684.733598 704.485018 733.636276 921.350903 []
40 3 1 200 rest 1 200 true 217.722397 16.593757 657.144743 695.158232 745.726065 916.803160 []
32 3 1 50 rest 1 200 true 218.817952 10.808913 689.809571 757.737985 867.650689 912.589694 []
56 3 4 200 rest 1 200 true 221.031876 9.197338 690.217169 711.800471 742.657817 903.072311 []
50 3 4 50 rest 4 200 true 221.263249 16.583482 688.637696 711.870214 781.197685 902.315850 []
58 3 4 200 rest 4 200 true 221.566956 11.037262 685.417461 713.923684 771.814053 901.132352 []
35 3 1 50 rest 4 200 false 225.719114 15.998348 704.701196 741.890962 852.664830 884.187996 []
33 3 1 50 rest 1 200 false 229.653366 9.844413 725.066803 775.186525 857.762245 869.461119 []
42 3 1 200 rest 4 200 true 231.016536 15.829218 737.382688 788.027859 885.482116 863.960992 []
49 3 4 50 rest 1 200 false 231.986927 11.193407 702.083677 769.889421 901.360146 860.495277 []
43 3 1 200 rest 4 200 false 239.150794 14.147647 722.982655 789.211063 929.436195 834.381347 []
34 3 1 50 rest 4 200 true 240.088790 121.078205 707.862815 771.405571 965.932529 831.402721 []
26 1 4 200 rest 4 200 true 413.608259 409.729690 442.576049 460.804621 502.762769 482.699096 []
17 1 4 50 rest 1 200 false 429.042835 412.423403 500.170846 522.423418 586.685379 465.431891 []
27 1 4 200 rest 4 200 false 432.609142 426.606234 488.443435 512.393140 556.238288 461.578501 []
25 1 4 200 rest 1 200 false 463.422714 450.181537 551.644801 602.270942 670.647806 430.891782 []
16 1 4 50 rest 1 200 true 475.510231 456.056479 583.716159 650.365364 746.791628 419.975983 []
19 1 4 50 rest 4 200 false 481.143061 450.734477 602.026223 689.302618 863.072782 414.795159 []
18 1 4 50 rest 4 200 true 488.185779 436.842244 628.922397 735.512654 1068.474298 408.992844 []
0 1 1 50 rest 1 200 true 489.269344 455.617128 612.294382 672.510108 832.322767 407.879172 []
24 1 4 200 rest 1 200 true 514.472545 488.358257 591.629431 631.392813 1517.062374 387.882855 []
1 1 1 50 rest 1 200 false 529.767457 514.151876 591.278115 621.463805 749.348556 376.649458 []
2 1 1 50 rest 4 200 true 547.618426 526.472215 661.947413 720.039676 863.596098 364.363839 []
10 1 1 200 rest 4 200 true 571.452398 556.699256 693.093315 751.197598 1024.233714 348.889260 []
11 1 1 200 rest 4 200 false 587.900216 556.869872 723.744376 774.244702 939.994423 339.396160 []
3 1 1 50 rest 4 200 false 593.880113 602.945695 737.993290 770.777543 1003.510371 336.075411 []
8 1 1 200 rest 1 200 true 633.043624 617.853285 741.229073 776.560578 1846.623159 314.908167 []
9 1 1 200 rest 1 200 false 641.530606 653.922529 802.558303 847.414484 1570.484029 310.839312 []

Deep dive on GRPC

[191]:
df[df["apiType"]=="grpc"].sort_values("rate", ascending=False)
[191]:
replicas server_workers server_threads apiType cpus max_wokers use_engine mean 50th 90th 95th 99th rate errors
60 3 4 200 grpc 1 200 true 31.389861 23.769589 71.583795 78.881398 91.312797 1586.593680 {'OK': 190361, 'Unavailable': 48}
52 3 4 50 grpc 1 200 true 31.398451 26.313000 64.841515 73.035800 88.744198 1586.555365 {'OK': 190333, 'Unavailable': 71}
45 3 1 200 grpc 1 200 false 32.191240 30.448302 60.616301 68.724406 91.484308 1547.003054 {'OK': 185606, 'Unavailable': 49}
61 3 4 200 grpc 1 200 false 32.727674 28.483400 63.750796 72.597310 90.693812 1521.590875 {'OK': 182555, 'Unavailable': 49}
55 3 4 50 grpc 4 200 false 33.629848 29.610701 67.065895 77.773100 97.296599 1479.320474 {'OK': 177471, 'Unavailable': 50}
47 3 1 200 grpc 4 200 false 33.861023 30.207400 70.272698 83.485103 105.639301 1469.503585 {'OK': 176302, 'Unavailable': 50}
62 3 4 200 grpc 4 200 true 34.746801 31.896585 72.732796 84.032763 99.433090 1432.045405 {'OK': 171799, 'Unavailable': 50}
54 3 4 50 grpc 4 200 true 34.786883 32.141197 72.554313 82.649702 95.049705 1430.209225 {'OK': 171578, 'Unavailable': 49}
37 3 1 50 grpc 1 200 false 35.149376 35.153187 62.842800 72.791800 94.240299 1416.745392 {'OK': 169973, 'Unavailable': 50}
36 3 1 50 grpc 1 200 true 35.167657 31.859300 65.644895 76.240799 98.925899 1415.967279 {'OK': 169880, 'Unavailable': 48}
46 3 1 200 grpc 4 200 true 35.286173 24.988500 83.079301 94.264796 111.448895 1410.595798 {'OK': 169202, 'Unavailable': 71}
53 3 4 50 grpc 1 200 false 35.543940 30.528900 69.449895 82.465882 100.381195 1400.945365 {'OK': 168074, 'Unavailable': 50}
63 3 4 200 grpc 4 200 false 35.706181 30.175300 76.121701 85.842385 99.072701 1393.469861 {'OK': 167180, 'Unavailable': 49}
39 3 1 50 grpc 4 200 false 36.026804 33.541192 69.942798 81.321704 108.528901 1381.482907 {'OK': 165711, 'Unavailable': 69}
38 3 1 50 grpc 4 200 true 36.325718 35.598498 73.211997 82.948302 102.248397 1369.739820 {'OK': 164333, 'Unavailable': 49}
44 3 1 200 grpc 1 200 true 37.326561 35.609388 70.522598 79.731401 101.297400 1334.058278 {'OK': 160053, 'Unavailable': 50}
29 1 4 200 grpc 1 200 false 63.240129 61.519201 72.905000 77.140700 89.520499 789.347786 {'OK': 94678, 'Unavailable': 50}
28 1 4 200 grpc 1 200 true 63.537119 61.855200 74.299100 79.876601 97.179900 785.631011 {'OK': 94233, 'Unavailable': 50}
20 1 4 50 grpc 1 200 true 65.711577 64.220500 78.085300 83.563600 94.907700 759.690398 {'OK': 91119, 'Unavailable': 50}
21 1 4 50 grpc 1 200 false 66.898143 63.420800 83.837100 92.332400 108.138499 746.209307 {'OK': 89501, 'Unavailable': 50}
30 1 4 200 grpc 4 200 true 67.211609 65.504200 79.989899 86.808200 106.460500 742.433252 {'OK': 89044, 'Unavailable': 50}
7 1 1 50 grpc 4 200 false 67.770632 62.168504 88.674303 102.537000 120.848185 736.385539 {'OK': 88318, 'Unavailable': 49}
31 1 4 200 grpc 4 200 false 70.577834 68.972899 84.869200 89.875600 102.761897 707.046156 {'OK': 84796, 'Unavailable': 50}
22 1 4 50 grpc 4 200 true 70.818411 67.591600 87.914104 97.004000 115.388900 704.647865 {'OK': 84514, 'Unavailable': 50}
15 1 1 200 grpc 4 200 false 71.571627 69.348700 91.609598 98.471998 111.237797 697.252435 {'OK': 83622, 'Unavailable': 50}
23 1 4 50 grpc 4 200 false 73.853780 70.604701 91.031400 98.064600 116.658902 675.704389 {'OK': 81035, 'Unavailable': 50}
14 1 1 200 grpc 4 200 true 89.662500 87.678702 107.762199 118.226099 146.838610 556.478774 {'OK': 66728, 'Unavailable': 50}
6 1 1 50 grpc 4 200 true 90.655025 91.964500 108.453597 116.581800 148.048199 550.406903 {'OK': 66003, 'Unavailable': 50}
5 1 1 50 grpc 1 200 false 92.930400 93.020601 113.056104 122.476104 150.119004 537.076992 {'OK': 64405, 'Unavailable': 50}
12 1 1 200 grpc 1 200 true 94.695951 94.988002 111.319799 118.210000 134.270997 527.054914 {'OK': 63202, 'Unavailable': 50}
4 1 1 50 grpc 1 200 true 95.322943 97.896699 117.221999 125.852400 141.615501 523.628160 {'OK': 62790, 'Unavailable': 50}
13 1 1 200 grpc 1 200 false 96.016296 97.410200 113.779899 120.184499 136.929395 519.810588 {'OK': 62332, 'Unavailable': 50}
[ ]: