Workload identity support for cosmosDB external scaler #80

vishal1997 · 2025-02-28T13:41:49Z

Provide a description of what has been changed

This PR introduces Managed Identity (MI) support for the Azure Cosmos DB external scaler and updates the Demo project to use MI-based access for autoscaling
Included detailed steps for configuring and deploying the Demo project with MI-based access.

Manual testing and validation:

Used the Demo app in the repo to generate data in cosmos DB, used MSI to access cosmos DB.
Used the Demo app in the repo to process the data generated in the first step, used MSI to access the cosmosDB.
Created the scale object having the clientId of MSI as properties, validated the pods count increased to 2 from 0 and back to 0 after processing all the data.

The manual testing was performed on AKS cluster, details to setup the cluster with MSI is added in the Readme file of the demo app.

Original PR: #78

Checklist

Commits are signed with Developer Certificate of Origin (DCO)
A PR is opened to update the documentation on our docs repo

Fixes #

Signed-off-by: Vighnesh Shenoy <[email protected]>

raorugan · 2025-03-03T06:51:39Z

src/Scaler.Demo/OrderProcessor/deploy.yaml

+            - name: CosmosDbConfig__LeaseContainerId
+              value: "OrderProcessorLeases"
+            - name: CosmosDbConfig__ProcessorName
+              value: "OrderProcessor"


Is it possible to share outcome of this test - order-processor with user-assigned and system assigned? Where was this tested on AKS or ACA?

Added the details in the description, this was tested on AKS cluster

krishnajaju · 2025-03-11T11:29:30Z

src/Scaler/deploy.yaml

  selector:
    matchLabels:
      app: cosmosdb-scaler
+      azure.workload.identity/use: "true" # Optional: Use Workload Identity for authentication. Refer https://azure.github.io/azure-workload-identity/docs/topics/service-account-labels-and-annotations.html for more details.


Do we want to enable workload identity by default in the scaler deployment?

Also, do we want to keep serviceAccountName as sa in scaler deployment? It might not clear on what this service account refers to until looking at the OrderProcessor Scaler Demo file.

We can probably use a different Scaler Deploy.yaml file for the Demo?

src/Scaler.Demo/README.md

src/Scaler.Demo/Shared/CosmosDbConfig.cs

krishnajaju · 2025-03-11T12:48:02Z

src/Scaler/Services/CosmosDbMetricProvider.cs

+            // Prioritize credential based connections
+            if (!string.IsNullOrWhiteSpace(metadata.Endpoint))
+            {
+                return _factory.GetCosmosClient(metadata.Endpoint, useCredentials: true, clientId: metadata.ClientId);


does passing null in clientId argument switch to authenticating system MI?

If the clientId here is null, the azure.workload.identity/client-id annotation in the service account will be picked as the default client ID during the client initialization.
This change does not support system managed identity, reason is we are using federated credentials for workload identity federation, which cannot be created for system assigned identity.
ref: https://azure.github.io/azure-workload-identity/docs/topics/federated-identity-credential.html

krishnajaju · 2025-03-11T12:52:03Z

src/Scaler/Services/CosmosDbFactory.cs

+        public CosmosClient GetCosmosClient(string endpointOrConnection, bool useCredentials, string clientId)
        {
-            return _cosmosClientCache.GetOrAdd(connection, CreateCosmosClient);
+            return _cosmosClientCache.GetOrAdd((endpointOrConnection, clientId), CreateCosmosClient(endpointOrConnection, useCredentials, clientId));


just to confirm, will clientId:null cause any issues in adding to dictionary?

The change is tested with null value.

krishnajaju · 2025-03-11T12:53:59Z

src/Scaler/Services/CosmosDbFactory.cs

+        /// </summary>
+        /// <param name="clientId">ClientId of the identity to be used. System identity is used if this is null. </param>
+        /// <returns></returns>
+        private TokenCredential GetChainedCredential(string clientId)


Did we validate change with both user-assigned and system MI?

Please refer this comment #80 (comment)

deploy/deploy-scaledobject-mi.yaml

krishnajaju · 2025-03-11T12:58:39Z

deploy/service-account-mi.yaml

+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: <service-account-name>  


why do we need this file here?

This is just a template for scaled-object using mi

JatinSanghvi

Added first set of review comments.

README.md

JatinSanghvi · 2025-03-14T11:09:09Z

README.md

+        # Managed Identity properties
+        endpoint: <endpoint>                                        # Optional. Account endpoint of the CosmosDB account containing the monitored container.
+        leaseEndpoint: <lease-endpoint>                             # Optional. Account endpoint of the CosmosDB account containing the lease container.
+        clientId: <client-Id>                                       # Optional. ClientId of the identity to be used. System assigned identity is used, if this is null.


Replace with `Client ID' here and in the description below. Should we say, 'if this is not provided'?

README.md

deploy/deploy-scaledobject-mi.yaml

deploy/service-account-mi.yaml

JatinSanghvi · 2025-03-14T11:40:51Z

src/Scaler.Demo/Shared/DemoHelper.cs

+        {
+            if (useCredentials)
+            {
+                var credential = GetChainedCredential(clientId);


Since CosmosClientOptions is being inlined, should be do the same for TokenCredential?

JatinSanghvi · 2025-03-14T11:44:00Z

src/Scaler.Demo/Shared/DemoHelper.cs

+        {
+            var options = new DefaultAzureCredentialOptions
+            {
+                ManagedIdentityClientId = clientId,


For my understanding, can clientId be null? Will that force it to use system assigned MSI?

Please refer this comment: #80 (comment)

src/Scaler.Demo/Shared/DemoHelper.cs

src/Scaler.Demo/OrderProcessor/sa.yaml

JatinSanghvi · 2025-03-14T11:54:40Z

src/Scaler.Demo/OrderGenerator/Program.cs

+        public static string _applicationName = "cosmosdb-order-generator";
+
        public static async Task Main(string[] args)
        {


Was it necessary to kill user interactivity and validation of input values? Can you explain the reason here. Let me know if this needs to be discussed offline.

Since this requires a drastic change, and in my opinion, it loses the charm of demo, we can think of keeping the order generator project connection string based only, as this was meant to be called locally, from demoing person's computer.

The intention here was that during the demo, we can keep generating new orders rapidly. I think the new demo would require updating the deployment YAML file and kubectl apply ing it, which will slow it down. The order processor is the main application that resembles user workload that requires KEDA scaling and the generator is there to have a complete end-to-end demo-able sample.

villanisaac-kr · 2025-11-10T23:13:32Z

Is there any ETA for this feature to be completed (Support for Workload identity)?

ni-bhandari · 2025-11-10T23:17:55Z

@villanisaac-kr currently working on completing this PR - #86. It has been approved by a reviewer, hoping to get it merged soon.

villanisaac-kr · 2025-11-10T23:35:59Z

@villanisaac-kr currently working on completing this PR - #86. It has been approved by a reviewer, hoping to get it merged soon.

Awesome thanks 🥇

Vighnesh Shenoy and others added 7 commits February 28, 2025 17:33

Update scaler metadata

a6c245a

Signed-off-by: Vighnesh Shenoy <[email protected]>

Update cosmos client to use credentials

98ccf42

Signed-off-by: Vighnesh Shenoy <[email protected]>

Update docs

14d5016

Signed-off-by: Vighnesh Shenoy <[email protected]>

Fix serialization and update tests

59f0d28

Signed-off-by: Vighnesh Shenoy <[email protected]>

fix MI client

be7d825

Update deployment file

590b247

Update demo project to use MSI

be9f135

vishal1997 requested review from JatinSanghvi and tomkerkhove as code owners February 28, 2025 13:41

v-shenoy mentioned this pull request Mar 3, 2025

Managed Identity + Workload Identity support. #78

Closed

2 tasks

raorugan reviewed Mar 3, 2025

View reviewed changes

Vishal Gupta added 4 commits March 3, 2025 13:09

Add back constring

57e8ba1

minor changes

ba9a0db

Update demo project

5115609

Update readme

757902d

vishal1997 force-pushed the cosmosdb_mi_changes branch from 9d715cb to 757902d Compare March 3, 2025 09:59

krishnajaju suggested changes Mar 11, 2025

View reviewed changes

JatinSanghvi reviewed Mar 14, 2025

View reviewed changes

vishal1997 force-pushed the cosmosdb_mi_changes branch from 11cb334 to edb6a0e Compare March 17, 2025 04:37

Address review comments

8fca784

vishal1997 force-pushed the cosmosdb_mi_changes branch from edb6a0e to 8fca784 Compare March 17, 2025 04:54

ni-bhandari mentioned this pull request Oct 2, 2025

Managed Identity Support for CosmosDB External Scaler #86

Merged

2 tasks

Workload identity support for cosmosDB external scaler #80

Are you sure you want to change the base?

Workload identity support for cosmosDB external scaler #80

Uh oh!

Conversation

vishal1997 commented Feb 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

raorugan Mar 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JatinSanghvi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

villanisaac-kr commented Nov 10, 2025

Uh oh!

ni-bhandari commented Nov 10, 2025

Uh oh!

villanisaac-kr commented Nov 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

vishal1997 commented Feb 28, 2025 •

edited

Loading

raorugan Mar 3, 2025 •

edited

Loading