This page was generated from examples/triton_gpt2/AzureSetup.ipynb.

Setup Azure Kubernetes Infrastructure

In this notebook we will - Login to Aure account - Create AKS cluster with - GPU enabled Spot VM nodepool for running ML elastic training - CPU VM nodepool for running typical workloads - Azure Storage Account for hosting model data - Deploy Kubernetes Components - Install Azure Blob CSI Driver to map Blob storage to container as persistent volumes - Create Kubernetes PersistentVolume and PersistentVolumeClaim

Define Variables

Set variables required for the project

subscription_id = "<xxxx-xxxx-xxxx-xxxx>"  # fill in
resource_group = "seldon"  # feel free to replace or use this default
region = "eastus2"  # ffeel free to replace or use this default

storage_account_name = "modeltestsgpt"  # fill in
storage_container_name = "gpt2tf"

aks_name = "modeltests"  # feel free to replace or use this default
aks_gpupool = "gpunodes"  # feel free to replace or use this default
aks_cpupool = "cpunodes"  # feel free to replace or use this default
aks_gpu_sku = "Standard_NC6s_v3"  # feel free to replace or use this default
aks_cpu_sku = "Standard_F8s_v2"

Azure account login

If you are not already logged in to an Azure account, the command below will initiate a login. This will pop up a browser where you can select your login. (if no web browser is available or if the web browser fails to open, use device code flow with az login --use-device-code or login in WSL command prompt and proceed to notebook)

[ ]:
az login -o table

[ ]:
!az account set --subscription "$subscription_id"
[ ]:
!az account show

Create Resource Group

Azure encourages the use of groups to organize all the Azure components you deploy. That way it is easier to find them but also we can delete a number of resources simply by deleting the group.

[ ]:
!az group create -l {region} -n {resource_group}

Create AKS Cluster and NodePools

Below, we create the AKS cluster with default 1 system node (to save time, in production use more nodes as per best practices) in the resource group we created earlier. This step can take 5 or more minutes.

[ ]:
!az aks create --resource-group {resource_group} \
    --name {aks_name} \
    --node-vm-size Standard_D8s_v3  \
    --node-count 1 \
    --location {region}  \
    --kubernetes-version 1.18.17 \
    --node-osdisk-type Ephemeral \

Connect to AKS Cluster

To configure kubectl to connect to Kubernetes cluster, run the following command

[ ]:
!az aks get-credentials --resource-group {resource_group} --name {aks_name}

Let’s verify connection by listing the nodes.

!kubectl get nodes
NAME                                STATUS   ROLES   AGE     VERSION
aks-agentpool-28613018-vmss000000   Ready    agent   28d     v1.19.9
aks-agentpool-28613018-vmss000001   Ready    agent   28d     v1.19.9
aks-agentpool-28613018-vmss000002   Ready    agent   28d     v1.19.9
aks-cpunodes-28613018-vmss000000    Ready    agent   28d     v1.19.9
aks-cpunodes-28613018-vmss000001    Ready    agent   28d     v1.19.9
aks-gpunodes-28613018-vmss000001    Ready    agent   5h27m   v1.19.9

Taint System node with CriticalAddonsOnly taint so it is available only for system workloads

[ ]:
!kubectl taint nodes -l CriticalAddonsOnly=true:NoSchedule --overwrite

Create GPU enabled and CPU Node Pools

To create GPU enabled nodepool, will use fully configured AKS image that contains the NVIDIA device plugin for Kubenetes, see Use the AKS specialized GPU image (preview). Creating nodepools could take five or more minutes.

[ ]:
!az feature register --name GPUDedicatedVHDPreview --namespace Microsoft.ContainerService
!az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/GPUDedicatedVHDPreview')].{Name:name,State:properties.state}"
!az provider register --namespace Microsoft.ContainerService
!az extension add --name aks-preview

Create GPU NodePool with GPU taint

For more information on Azure Nodepools

print ({aks_gpu_sku})
!az aks nodepool add \
    --resource-group {resource_group} \
    --cluster-name {aks_name} \
    --name {aks_gpupool} \
    --node-taints \
    --node-count 1 \
    --node-vm-size  {aks_gpu_sku} \
    --aks-custom-headers UseGPUDedicatedVHD=true,usegen2vm=true
The behavior of this command has been altered by the following extension: aks-preview
Node pool gpunodes already exists, please try a different name, use 'aks nodepool list' to get current list of node pool
CPU times: user 275 ms, sys: 79 ms, total: 354 ms
Wall time: 5.38 s

Verify GPU is available on Kubernetes Node

Now use the kubectl describe node command to confirm that the GPUs are schedulable. Under the Capacity section, for Standard_NC12 sku the GPU should list as 2

!kubectl describe node -l accelerator=nvidia | grep nvidia -A 5 -B 5
Name:               aks-gpunodes-28613018-vmss000001
Roles:              agent
Labels:             accelerator=nvidia
  cpu:                            12
  ephemeral-storage:              129900528Ki
  hugepages-1Gi:                  0
  hugepages-2Mi:                  0
  memory:                         115387540Ki                 2
  pods:                           30
  attachable-volumes-azure-disk:  48
  cpu:                            11780m
  ephemeral-storage:              119716326407
  hugepages-1Gi:                  0
  hugepages-2Mi:                  0
  memory:                         105854100Ki                 2
  pods:                           30
System Info:
  Machine ID:                 db67bd967e1441febad873ba49d35adc
  System UUID:                f39ce4bc-11c6-8643-8a8a-dfb4998a0524
  Boot ID:                    eb926e42-d4e7-4760-b124-9b09c0e56c57
  memory                         275Mi (0%)  850Mi (0%)
  ephemeral-storage              0 (0%)      0 (0%)
  hugepages-1Gi                  0 (0%)      0 (0%)
  hugepages-2Mi                  0 (0%)      0 (0%)
  attachable-volumes-azure-disk  0           0                 0           0
Events:                          <none>

Create CPU NodePool for running regular workloads

!az aks nodepool add \
  --resource-group {resource_group} \
    --cluster-name {aks_name} \
    --name {aks_cpupool} \
    --enable-cluster-autoscaler \
    --node-osdisk-type Ephemeral \
    --min-count 1 \
    --max-count 3 \
    --node-vm-size {aks_cpu_sku}  \
    --node-osdisk-size 128
The behavior of this command has been altered by the following extension: aks-preview
  "agentPoolType": "VirtualMachineScaleSets",
  "availabilityZones": null,
  "count": 3,
  "enableAutoScaling": true,
  "enableEncryptionAtHost": false,
  "enableFips": false,
  "enableNodePublicIp": false,
  "gpuInstanceProfile": null,
  "id": "/subscriptions/xxxx-xxxx-xxxx-xxxx-xxxxxx/resourcegroups/seldon/providers/Microsoft.ContainerService/managedClusters/modeltests/agentPools/cpunodes",
  "kubeletConfig": null,
  "kubeletDiskType": "OS",
  "linuxOsConfig": null,
  "maxCount": 3,
  "maxPods": 30,
  "minCount": 1,
  "mode": "User",
  "name": "cpunodes",
  "nodeImageVersion": "AKSUbuntu-1804gen2containerd-2021.05.08",
  "nodeLabels": null,
  "nodePublicIpPrefixId": null,
  "nodeTaints": null,
  "orchestratorVersion": "1.19.9",
  "osDiskSizeGb": 128,
  "osDiskType": "Ephemeral",
  "osSku": "Ubuntu",
  "osType": "Linux",
  "podSubnetId": null,
  "powerState": {
    "code": "Running"
  "provisioningState": "Succeeded",
  "proximityPlacementGroupId": null,
  "resourceGroup": "seldon",
  "scaleSetEvictionPolicy": null,
  "scaleSetPriority": null,
  "spotMaxPrice": null,
  "tags": null,
  "type": "Microsoft.ContainerService/managedClusters/agentPools",
  "upgradeSettings": {
    "maxSurge": null
  "vmSize": "Standard_F8s_v2",
  "vnetSubnetId": "/subscriptions/xxxxx-xxxx-xxxx-xxxxx-xxxxxx/resourceGroups/seldon/providers/Microsoft.Network/virtualNetworks/seldon-vnet/subnets/default"
CPU times: user 4.17 s, sys: 1.51 s, total: 5.68 s
Wall time: 2min 36s

Verify Taints on the Kubernetes nodes

Verify that system pool and have the Taints CriticalAddonsOnly and sku=gpu respectively

!kubectl get nodes -o json | jq '.items[].spec.taints'
    "effect": "NoSchedule",
    "key": "CriticalAddonsOnly",
    "value": "true"
    "effect": "NoSchedule",
    "key": "CriticalAddonsOnly",
    "value": "true"
    "effect": "NoSchedule",
    "key": "CriticalAddonsOnly",
    "value": "true"
    "effect": "NoSchedule",
    "key": "sku",
    "value": "gpu"

Create Storage Account for training data

In this section of the notebook, we’ll create an Azure blob storage that we’ll use throughout the tutorial. This object store will be used to store input images and save checkpoints. Use az cli to create the account

!az storage account create -n {storage_account_name} -g {resource_group} --query 'provisioningState'
CPU times: user 674 ms, sys: 214 ms, total: 888 ms
Wall time: 22 s

Grab the keys of the storage account that was just created.We would need them for binding Kubernetes Persistent Volume. The –quote ‘[0].value’ part of the command simply means to select the value of the zero-th indexed of the set of keys.

key = !az storage account keys list --account-name {storage_account_name} -g {resource_group} --query '[0].value' -o tsv

The stdout from the command above is stored in a string array of 1. Select the element in the array.

storage_account_key = key[0]
# create storage container

!az storage container create \
    --account-name {storage_account_name} \
    --account-key {storage_account_key} \
    --name {storage_container_name}
  "created": true

Install Kubernetes Blob CSI Driver

Azure Blob Storage CSI driver for Kubernetes allows Kubernetes to access Azure Storage. We will deploy it using Helm3 package manager as described in the docs

[ ]:
!az aks get-credentials --resource-group {resource_group} --name {aks_name}
!helm repo add blob-csi-driver
!helm install blob-csi-driver blob-csi-driver/blob-csi-driver --namespace kube-system --version v1.1.0
"blob-csi-driver" already exists with the same configuration, skipping
W0527 23:11:20.183604   13719 warnings.go:70] CSIDriver is deprecated in v1.19+, unavailable in v1.22+; use CSIDriver
W0527 23:11:20.506450   13719 warnings.go:70] CSIDriver is deprecated in v1.19+, unavailable in v1.22+; use CSIDriver
NAME: blob-csi-driver
LAST DEPLOYED: Thu May 27 23:11:19 2021
NAMESPACE: kube-system
STATUS: deployed
The Azure Blob Storage CSI driver is getting deployed to your cluster.

To check Azure Blob Storage CSI driver pods status, please run:

  kubectl --namespace=kube-system get pods --selector="release=blob-csi-driver" --watch
!kubectl -n kube-system get pods -l ""
NAME                                   READY   STATUS    RESTARTS   AGE
csi-blob-controller-7b9db4967c-fbsm2   4/4     Running   0          22s
csi-blob-controller-7b9db4967c-hdglw   4/4     Running   0          22s
csi-blob-node-7tgl8                    3/3     Running   0          22s
csi-blob-node-89rkn                    3/3     Running   0          22s
csi-blob-node-nnhfh                    3/3     Running   0          22s
csi-blob-node-pb584                    3/3     Running   0          22s
csi-blob-node-q6z6t                    3/3     Running   0          22s
csi-blob-node-tq4mh                    3/3     Running   0          22s

Create Persistent Volume for Azure Blob

For more details on creating PersistentVolume using CSI driver refer to

# Create secret to access storage account
!kubectl create secret generic azure-blobsecret --from-literal azurestorageaccountname={storage_account_name} --from-literal azurestorageaccountkey="{storage_account_key}" --type=Opaque
secret/azure-blobsecret created

Persistent Volume YAML definition is in azure-blobfules-pv.yaml with fields pointing to secret created above and containername we created in storage account:

  readOnly: false
  volumeHandle: trainingdata  # make sure this volumeid is unique in the cluster
    containerName: workerdata # !! Modify if changed in Notebook
    name: azure-blobsecret
%%writefile azure-blobfuse-pv.yaml
apiVersion: v1
kind: PersistentVolume
  name: pv-gptblob

    storage: 10Gi
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain  # "Delete" is not supported in static provisioning
    readOnly: false
    volumeHandle: trainingdata  # make sure this volumeid is unique in the cluster
      containerName: gpt2onnx # Modify if changed in Notebook
      name: azure-blobsecret
      namespace: default
    - -o uid=8888     # user in  Pod security context
    - -o allow_other

kind: PersistentVolumeClaim
apiVersion: v1
  name: pvc-gptblob

    - ReadWriteMany
      storage: 10Gi
  volumeName: pv-gptblob
  storageClassName: ""

Overwriting azure-blobfuse-pv.yaml
# Create PersistentVolume and PersistenVollumeClaim for container mounts
!kubectl apply -f  azure-blobfuse-pv.yaml
persistentvolume/pv-gptblob created
persistentvolumeclaim/pvc-gptblob created
# Verify PVC is bound
!kubectl get pv,pvc
NAME                          CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS        CLAIM                 STORAGECLASS   REASON   AGE
persistentvolume/pv-blob      10Gi       RWX            Retain           Terminating   default/pvc-blob                              113m
persistentvolume/pv-gptblob   10Gi       RWX            Retain           Bound         default/pvc-gptblob                           18s

NAME                                STATUS        VOLUME       CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/pvc-blob      Terminating   pv-blob      10Gi       RWX                           113m
persistentvolumeclaim/pvc-gptblob   Bound         pv-gptblob   10Gi       RWX                           17s

In the end of this step you will have AKS cluster and Storage account in resource group. ALK cluster will have cpu and gpu nodepools in addition to system nodepool.