Scale Kubernetes pods based on Service Bus Queue using Keda

In this blog post, we will use KEDA (Kubernetes Event-driven Autoscaling) for autoscaling pod count based on Azure Service Bus Queue length. I will be using the local Kubernetes cluster but it should work with Azure Kubernetes Service (AKS) also. All code used in this blog post can be found https://github.com/lets-learn-it/keda-examples/tree/master/azure-service-bus-queue.

Plan of Action as below,

Deploy KEDA in Kubernetes cluster
Create Azure Service Bus Queue using Terraform
Writing Kubernetes configuration files
Testing

Deploy KEDA

KEDA can be installed in the Kubernetes cluster using Helm. More info https://keda.sh/docs/2.6/deploy/. I deployed KEDA 2.6 in Kubernetes 1.22.x.

Azure Service Bus Queue

We need to create a resource group in order to create a service bus namespace & queue inside that namespace. Below is terraform code to achieve that.

resource "azurerm_resource_group" "kedaq-rg" {
  name     = "keda-demo-rg"
  location = "West Europe"
}

resource "azurerm_servicebus_namespace" "keda-namespace" {
  name                = var.servicebus-namespace-name
  location            = azurerm_resource_group.kedaq-rg.location
  resource_group_name = azurerm_resource_group.kedaq-rg.name
  sku                 = "Standard"

}

resource "azurerm_servicebus_queue" "keda-demoq" {
  name                = var.servicebus-queue-name
  namespace_name      = azurerm_servicebus_namespace.keda-namespace.name
  resource_group_name = azurerm_resource_group.kedaq-rg.name

  enable_partitioning = true
}

For the service bus namespace, we can use the default shared access policy but for the queue, we need to create one access policy. & make sure to create manage access policy (as per KEDA docs).

resource "azurerm_servicebus_queue_authorization_rule" "queuerule" {
  name                = "queuerule"
  namespace_name      = azurerm_servicebus_namespace.keda-namespace.name
  queue_name          = azurerm_servicebus_queue.keda-demoq.name
  resource_group_name = azurerm_resource_group.kedaq-rg.name

  # As per KEDA docs,
  # Service Bus Shared Access Policy needs to be of type Manage.
  # Manage access is required for KEDA to be able to get metrics from Service Bus.
  manage = true
  listen = true
  send   = true
}

make sure to declare variables used in the above configuration & also need to do output some values i.e. connection strings.

output "namespace_primary_connection_string" {
    value     = azurerm_servicebus_namespace.keda-namespace.default_primary_connection_string
    sensitive = true
}

output "queue_primary_connection_string" {
    value     = azurerm_servicebus_queue_authorization_rule.queuerule.primary_connection_string
    sensitive = true
}

Create resources

Run plan & apply to create resources in Azure. After running terraform, I got 1 Azure service bus namespace & one queue inside it.

Kubernetes configuration files

I will deploy Nginx as a deployment which will do nothing. We will add messages & remove messages from the queue manually to test scaling.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-app
spec:
  selector:
    matchLabels:
      app: demo-app
  template:
    metadata:
      labels:
        app: demo-app
    spec:
      containers:
      - name: demo-app
        image: nginx
        resources:
          limits:
            memory: "128Mi"
            cpu: "100m"
        ports:
        - containerPort: 8080
        env:
          # This is not required when using triggerauthentication object
          - name: keda-secret
            valueFrom: 
              secretKeyRef:
                name: queue-policy-secret
                key: connection-string

Everything in the above YAML is normal except the environment variable keda-secret. We can authorize KEDA ScaledObject to access our queue in multiple ways. One of the ways needs this secret.

We will see 2 ways to authorize KEDA to access the queue. There is another way of using identity but for simplicity, we will not see this way in this post.

Way 01 Use TriggerAuthentication

First create secret using default primary connection string of Azure service bus namespace, like below

apiVersion: v1
kind: Secret
metadata:
  name: namespace-secret
type: Opaque
data:
  # This is azure service bus namespace connection string
  connection: RW5kcG9pbnQ9c2I6Ly9rZWRhLXNlcnZpY2VidXMtbmFtZXNwYWNlLnNlcnZpY2VidXMud2luZG93cy5uZXQvO1NoYXJlZEFjY2Vzc0tleU5hbWU9Um9vdE1hbmFnZVNoYXJlZEFjY2Vzc0tleTtTaGFyZWRBY2Nlc3NLZXk9bGQ4akIyV042SWRlbzJkMWJobXpYR01SZWRIOXg5ZloremFxSGtmVUQrcz0=

Now, once we have a secret, we can create TriggerAuthentication for that secret.

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: azure-servicebus-auth
spec:
  secretTargetRef:
    - key: connection
      name: namespace-secret # name of secret
      parameter: connection # key in secret

The remaining piece in the puzzle is ScaledObject. Which we can create using below YAML,

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: azure-servicebus-queue-scaledobject
  namespace: default
spec:
  scaleTargetRef:
    kind: Deployment
    name: demo-app
  pollingInterval: 30
  cooldownPeriod: 60
  minReplicaCount: 1
  maxReplicaCount: 4

  triggers:
  - type: azure-servicebus
    metadata:
      queueName: keda-demoq
      messageCount: "5"
    authenticationRef:
        # reference to TriggerAuthentication
        name: azure-servicebus-auth

That's it, It will create HPA & if HPA is not created then check the logs of the operator. Make sure to check status of ScaledObject.

Testing

I sent 6 messages to the queue using the service bus explorer. And KEDA up scaled pods to 2. To check to downscale, remove some messages from the queue (make sure to wait for cooldownPeriod).

Way 02 Use pod's environment variable

We can use the pod's environment variable for authorization also. I personally discourage using it. In this method, we need the connection string of the queue as an environment variable of deployment. But before starting way 02, remove resources from way 01.

The secret will get changed (not connection string of namespace but of the queue itself. for this reason, we created authorization rule in terraform). And we don't need TriggeredAuthentication.

apiVersion: v1
kind: Secret
metadata:
  name: queue-policy-secret
type: Opaque
data:
  # This is queue connection string
  connection: RW5kcG9pbnQ9c2I6Ly9rZWRhLXNlcnZpY2VidXMtbmFtZXNwYWNlLnNlcnZpY2VidXMud2luZG93cy5uZXQvO1NoYXJlZEFjY2Vzc0tleU5hbWU9cXVldWVydWxlO1NoYXJlZEFjY2Vzc0tleT1pWTNzSlh5OWorL0wvZVhMYmU5RmFseWVDU1pMWHM4WDFQVTZ2dkhzeW1nPTtFbnRpdHlQYXRoPWtlZGEtZGVtb3E=

The only change in ScaledObject is connectionFromEnv parameter & removing TriggerAuthentication reference.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: azure-servicebus-queue-scaledobject
  namespace: default
spec:
  scaleTargetRef:
    kind: Deployment
    name: demo-app
  pollingInterval: 30
  cooldownPeriod: 60
  minReplicaCount: 1
  maxReplicaCount: 4

  triggers:
  - type: azure-servicebus
    metadata:
      queueName: keda-demoq
      messageCount: "5"
      # ENV var of deployment
      connectionFromEnv: keda-secret

This time, you can't see anything in front of Authentication but it should show Ready status & it will work.

Cleanup

Make sure to clean all resources, especially those you created in Azure. No one wants to pay the cost for unused resources.

References

[1] https://keda.sh/docs/2.6/scalers/azure-service-bus/
[2] https://github.com/kedacore/keda/blob/main/pkg/scalers/azure_servicebus_scaler.go